Log anomaly detection method, device and equipment and storage medium

Abstract

The application provides a log anomaly detection method, device and equipment and a storage medium, comprising: obtaining target log texts of a target system; inputting each target log text into a log template mapping model to obtain each mapping log template; mapping each mapping log template into target log label information to form each log label sequence; slidingly intercepting each log label sequence based on a sliding window to input a log label subset of any sliding window into a log detection model to obtain a detection result of any sliding window; and determining a final log detection result based on the detection result of any sliding window and next target log label information of the sliding window. The application determines the mapping log template of the target log text by using the log template mapping model, ensures the uniqueness of log text classification, and detects by using the log detection model based on the mapping log template, without manual analysis of logs, thereby improving the efficiency and accuracy of log anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for detecting log anomalies. Background Technology

[0002] Logs are valuable information representing the operational status of a system. Currently, commonly used methods for detecting log anomalies mainly include the following: The first method involves extracting log sequences of abnormal system states from a large log set. Testers then manually analyze these sequences and write test cases to detect system log anomalies. However, manual analysis is inefficient for massive amounts of system logs. The second method involves pre-acquiring a large amount of historical logs and then using clustering algorithms to process them to build a log detection model, which is then used to detect log anomalies. However, the clustering results are highly dependent on the dataset, which can lead to different clustering results for the same log text, resulting in lower accuracy in log detection. Summary of the Invention

[0003] This invention provides a log anomaly detection method, apparatus, device, and storage medium, aiming to improve the accuracy and efficiency of log anomaly detection.

[0004] This invention provides a log anomaly detection method, comprising:

[0005] Obtain several target log texts corresponding to the target system;

[0006] Each of the target log texts is input into the log template mapping model to obtain the mapped log template corresponding to each target log text output by the log template mapping model;

[0007] Each of the mapped log templates is mapped to target log tag information, forming multiple log tag sequences;

[0008] Based on a pre-set sliding window, any of the log tag sequences is slid-trimmed to input the log tag subset corresponding to any sliding window into the log detection model, thereby obtaining the detection result corresponding to any sliding window output by the log detection model.

[0009] For any detection result corresponding to a sliding window, the final log detection result is determined based on the detection result and the next target log tag information corresponding to the sliding window;

[0010] The log template mapping model is obtained through iterative training based on each structured log text and the corresponding log template to be trained for each structured log text; the log template to be trained is obtained by clustering the structured log text.

[0011] The log detection model is obtained by iterative training based on each training flag sequence; the training flag sequence is obtained by mapping each training log template.

[0012] According to the log anomaly detection method provided by the present invention, the log template mapping model is trained based on the following steps:

[0013] Obtain the log codes generated during the operation of the target system;

[0014] Based on the log codes described above, several original log texts are selected from the historical log files;

[0015] The original log texts are processed into structured log texts to obtain structured log texts.

[0016] Clustering is performed on each of the structured log texts to obtain each log template to be trained;

[0017] Based on the structured log texts and the log templates to be trained, the initial mapping model is iteratively trained to obtain the log template mapping model.

[0018] According to a log anomaly detection method provided by the present invention, after performing clustering processing on each of the structured log texts to obtain each log template to be trained, the method further includes:

[0019] Each of the log templates to be trained is processed by string conversion to obtain the hash value corresponding to each log template to be trained.

[0020] Configure the log tag information corresponding to each hash value;

[0021] Construct a mapping relationship between each of the log templates to be trained and each of the log tag information, wherein the mapping relationship is used to map the mapped log templates to the target log tag information.

[0022] According to the log anomaly detection method provided by the present invention, the log anomaly detection module is trained based on the following steps:

[0023] The structured log texts are grouped to obtain several sets of historical log sequences;

[0024] Based on the mapping relationship, the training log templates corresponding to each structured log text in each of the historical log sequences are mapped to obtain the mapping label information corresponding to each of the historical log sequences.

[0025] Based on the mapping label information corresponding to each of the historical log sequences, each training flag sequence is formed;

[0026] Based on the aforementioned training flag sequences, the training detection model is iteratively trained to obtain the log detection model.

[0027] According to a log anomaly detection method provided by the present invention, the step of iteratively training the detection model to be trained based on each of the training flag sequences to obtain the log detection model includes:

[0028] For any training label sequence, the sliding window is used to slide and extract the mapping label information in the training label sequence to obtain a sequence subset corresponding to each sliding window;

[0029] Each of the sequence subsets corresponding to the sliding window is used as a training sample, and the next mapping label information corresponding to the sliding window is used as the sample label of the training sample.

[0030] Based on multiple training samples and their corresponding sample labels, the detection model to be trained is iteratively trained to obtain the log detection model.

[0031] According to a log anomaly detection method provided by the present invention, the step of mapping each of the mapped log templates to target log tag information to form multiple log tag sequences includes:

[0032] The target log texts are grouped to obtain log group sequences;

[0033] According to the mapping relationship, the mapping log template corresponding to each target log text in any log grouping sequence is mapped to the target log tag information;

[0034] Each log tag sequence is formed based on the target log tag information corresponding to any of the log grouping sequences.

[0035] According to a log anomaly detection method provided by the present invention, the step of determining a final log detection result based on the detection result corresponding to any sliding window and the next target log tag information corresponding to the sliding window includes:

[0036] The detection result is matched with the next target log tag information in the sliding window;

[0037] If the match is successful, the final log detection result is determined to be in a normal state;

[0038] If the match fails, the final log detection result is determined to be an abnormal state.

[0039] The present invention also provides a log anomaly detection device, comprising:

[0040] The acquisition module is used to acquire several target log texts corresponding to the target system;

[0041] The mapping module is used to input each of the target log texts into the log template mapping model to obtain the mapped log templates corresponding to each target log text output by the log template mapping model.

[0042] A forming module is used to map each of the mapped log templates to target log tag information, thereby forming multiple log tag sequences;

[0043] The detection module is used to slide and truncate any of the log tag sequences based on a pre-set sliding window, so as to input the log tag subset corresponding to any sliding window into the log detection model and obtain the detection result corresponding to any sliding window output by the log detection model.

[0044] The determination module is used to determine the final log detection result based on the detection result corresponding to any sliding window and the next target log tag information corresponding to the sliding window.

[0045] The log template mapping model is obtained through iterative training based on each structured log text and the corresponding log template to be trained for each structured log text; the log template to be trained is obtained by clustering the structured log text.

[0046] The log detection model is obtained by iterative training based on each training flag sequence; the training flag sequence is obtained by mapping each training log template.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the log anomaly detection method as described above.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log anomaly detection method as described above.

[0049] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the log anomaly detection method as described above.

[0050] The log anomaly detection method, apparatus, device, and storage medium provided by this invention construct a log template mapping model based on various structured log texts and training log templates obtained by clustering these structured log texts. This model predicts the mapped log templates corresponding to each target log text, ensuring that the log templates corresponding to the same log text are uniquely determined. Furthermore, based on the mapped log templates, multiple log tag sequences are formed, enabling detection using the log detection model. This eliminates the need for manual log analysis, effectively improving the efficiency and accuracy of log anomaly detection. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is one of the flowcharts of the log anomaly detection method provided by the present invention;

[0053] Figure 2 This is the second flowchart of the log anomaly detection method provided by the present invention;

[0054] Figure 3 This is the third flowchart of the log anomaly detection method provided by the present invention;

[0055] Figure 4 This is the fourth flowchart of the log anomaly detection method provided by the present invention;

[0056] Figure 5 This is a schematic diagram of the log anomaly detection device provided by the present invention;

[0057] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0059] The terminology used in one or more embodiments of the present invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention. The singular forms “a,” “the,” and “the” used in one or more embodiments of the invention are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of the invention refers to and includes any or all possible combinations of one or more associated listed items.

[0060] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of the present invention, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of the present invention, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "when".

[0061] Figure 1 This is one of the flowcharts illustrating the log anomaly detection method provided by this invention. For example... Figure 1 As shown, the log anomaly detection method includes:

[0062] Step 11: Obtain several target log texts corresponding to the target system;

[0063] It should be noted that the text content in the log files is unstructured text strings. Different log files have different content formats. Log content can be analyzed to extract data structures, thereby identifying the structured fields within the logs. Specifically, the process first involves obtaining the various log texts to be processed generated by the target system during its operation. Then, these log texts are processed to obtain the target log texts, for example, using regular expressions for structuring.

[0064] Step 12: Input each of the target log texts into the log template mapping model to obtain the mapped log templates corresponding to each target log text output by the log template mapping model; wherein, the log template mapping model is obtained by iterative training based on each structured log text and the log template to be trained corresponding to each structured log text; the log template to be trained is obtained by clustering the structured log texts;

[0065] Specifically, for any target log text, the target log text is input into the log template mapping model, and then the mapped log template corresponding to the target log text is determined based on the output of the log template mapping model. Understandably, the log template mapping model is obtained through iterative training based on each structured log text and its corresponding training log template. It should be noted that the structured log text is obtained by structuring the original log text, and the training log template is obtained by clustering the structured log text, such that any structured log text is configured with its associated training log template.

[0066] It should also be noted that although structured log text can be clustered into corresponding log templates, the same log text may yield different log templates using the clustering algorithm, which can affect the log detection results and lead to lower accuracy. Therefore, in this embodiment, a log template mapping model is trained based on the structured log text and the corresponding log template. This model can then be used to map each log text to obtain a mapped log template, ensuring that the mapped log template for the same log text is uniquely determined.

[0067] Step 13: Map each of the mapped log templates to target log tag information to form multiple log tag sequences;

[0068] Specifically, firstly, according to the business logic, the target log texts are grouped to obtain each log group sequence. Then, based on each log group sequence and the mapping log template corresponding to each target log text, multiple template sequences composed of each mapping log template are determined.

[0069] Further, in one embodiment, ID tags are pre-configured for each log template, and the ID tags of each log template form a log tag sequence corresponding to each template sequence. In another embodiment, each log template to be trained is subjected to string conversion processing, for example, MD5 processing, to obtain a hash value corresponding to each log template to be trained. Further, log tag information corresponding to each hash value is configured, wherein the log tag information can be used as a category tag for the log template, thereby constructing a mapping relationship between each log template to be trained and each log tag information. Thus, according to the mapping relationship, the mapped log templates in each template sequence can be mapped to target log tag information, and each log tag sequence is formed based on the target log tag information corresponding to any template sequence.

[0070] Step 14: Based on a pre-set sliding window, any of the log tag sequences is slid-trimmed to input the log tag subset corresponding to any sliding window into the log detection model, and the detection result corresponding to any sliding window output by the log detection model is obtained.

[0071] The log detection model is obtained through iterative training based on each training flag sequence; the training flag sequence is obtained by mapping each training log template.

[0072] Specifically, the following steps are performed for any sequence of log tags:

[0073] Based on a pre-set sliding window, the log tag sequence is truncated to obtain multiple log tag subsets corresponding to the sliding windows. The sliding window can be set according to actual conditions; preferably, it is set to an integer multiple of 10. Further, the log tag subset corresponding to any sliding window is input into the log detection model to obtain the detection result corresponding to any sliding window output by the log detection model. Understandably, the log detection model is obtained through iterative training of each training label sequence. The training label sequence is obtained by mapping each training log template according to a mapping relationship, thus enabling log detection processing using the log detection model. The detection result includes the probability values ​​of a preset number of label detection results. Optionally, a preset number of label detection results are output according to the magnitude of the probability values. The preset number can be set according to actual conditions. In this embodiment, to improve the accuracy of anomaly detection, the preset number is set to multiple, for example, outputting 5 label detection results with higher probability values.

[0074] For example, if the log tag sequence is [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] and the sliding window is 10, the subset of log tags [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] is input into the log detection model to obtain the detection result corresponding to the sliding window.

[0075] Step 15: For any detection result corresponding to a sliding window, determine the final log detection result based on the detection result and the next target log tag information corresponding to the sliding window.

[0076] Specifically, the following steps are performed for the detection result corresponding to any sliding window:

[0077] Each label detection result in the detection results is matched with the next target log label information corresponding to the sliding window. For example, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], the sliding window is 10, and the first sliding window is a subset of log labels [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. The next target log label information corresponding to this sliding window is 11. Further, if the match fails, it proves that the log corresponding to the next target log label information is abnormal, and thus the final log detection result is determined to be an abnormal state. If the match succeeds, it proves that the log corresponding to the next target log label information is normal, and thus the final log detection result is determined to be normal.

[0078] This invention constructs a log template mapping model based on various structured log texts and training log templates obtained by clustering these structured log texts. This model can then predict the mapped log template for each target log text, ensuring that the log template for the same log text is uniquely determined. Furthermore, based on the mapped log templates, multiple log tag sequences are formed, which can then be used for detection by a log detection model, eliminating the need for manual log analysis and effectively improving the efficiency and accuracy of log anomaly detection.

[0079] Figure 2 This is the second flowchart illustrating the log anomaly detection method provided by this invention. For example... Figure 2 As shown, in one embodiment of the present invention, step 13 above: mapping each of the mapped log templates to target log tag information to form multiple log tag sequences, includes:

[0080] The target log texts are grouped to obtain log group sequences; according to the mapping relationship, the mapping log templates corresponding to each target log text in any log group sequence are mapped to the target log tag information; based on the target log tag information corresponding to any log group sequence, each log tag sequence is formed.

[0081] Specifically, according to the business logic of the target system, the target log texts are grouped to obtain various log grouping sequences. For example, when the microservices are in a distributed architecture, they are grouped sequentially according to cluster-node-container-microservice-function-thread; for log detection of a single microservice, they can be grouped according to a hierarchical structure such as container-microservice-function-thread. Further, according to the mapping relationship between log templates and log tag information, the mapping log template corresponding to each target log text in any log grouping sequence is mapped to the target log tag information corresponding to the mapping log template, thereby forming various log tag sequences based on the target log tag information corresponding to any log grouping sequence.

[0082] This invention converts log grouping sequences into log tag sequences, thereby enabling the log detection model to detect the log tag sequences. The log tag sequences have a shorter data type length, which improves the efficiency of model detection.

[0083] In one embodiment of the present invention, the log template mapping model is trained based on the following steps:

[0084] Step 21: Obtain the various log codes generated during the operation of the target system;

[0085] It should be noted that Log4J is a commonly used logging component that can record the running status of different business functions. Log4J is an open-source project of Apache. It is a logging operation package. Using Log4J, you can specify the destination of log information output, such as the console, file, GUI components and NT event loggers; you can also control the output format of each log.

[0086] Specifically, the runtime code file of the target system is obtained, and then all log output packets are searched in the runtime code file. Based on each log output packet, the code of all output logs is searched and used as the target log code. The output log code includes codes such as INFO, ERROR, and WARNING.

[0087] Step 22: Based on the log codes, select several original log texts from the historical log files;

[0088] Specifically, log texts covering all log codes are selected from the currently saved historical log files to obtain several original log texts. In another embodiment, when the number of original log texts is large, the similarity between each original log text and the log code is calculated using a preset similarity calculation algorithm. The preset similarity calculation algorithm includes algorithms such as Jaccard similarity, Levenshtein edit distance, Simhash, TF-IDF, and BM25. Preferably, the TF-IDF algorithm is selected. Further, a preset number of original log texts with high similarity are selected. The preset number of texts can be set according to actual conditions and is not specifically limited here.

[0089] Step 23: Perform structuring processing on each of the original log texts to obtain each structured log text;

[0090] Specifically, based on the text format corresponding to each of the original log texts, regular expressions are used to perform structured processing on each of the original log texts to obtain each structured log text. For example, the original log text format is year-month-day hour:minute:second, log type, log level---[container name, thread name]function name:log content; the corresponding regular expression format is... <date> <time> <type> <level> <container> <task> <func> : <content>.

[0091] Step 24: Cluster the structured log texts to obtain the log templates to be trained.

[0092] It should be noted that each log template to be trained corresponds to a type of log text. Specifically, according to a preset log parsing algorithm, the structured log texts are clustered to obtain each log template to be trained. The preset log parsing algorithms include the Drain algorithm and the Logram algorithm, etc., and preferably, the Drain algorithm is selected. For example, the log template to be trained corresponding to "Start to send the 10msg" is "Start to send the <*>msg"; the log template to be trained corresponding to "Sending msg service start" is "<*>service start".

[0093] Step 25: Based on the structured log texts and the log templates to be trained, the initial mapping model is iteratively trained to obtain the log template mapping model.

[0094] It should be noted that the initial mapping model includes time-series models such as RNN, LSTM, and Transformer. To improve data processing efficiency, in this embodiment, the Transformer time-series model is preferably selected. The Transformer model can be applied to tasks such as translation and text summarization. The Transformer model architecture allows for more parallel computation, enabling the processing of all input data at once, thereby reducing training time.

[0095] Specifically, for any structured log text, the structured log text is input into an initial mapping model to obtain the predicted value output by the initial mapping model. Then, based on the log template to be trained corresponding to the structured log text and the predicted value, the model loss value is calculated. Then, based on the model loss value, the parameters of the initial mapping model are updated. This model training ends, and the next model training is performed until a preset training termination condition is reached to obtain the log template mapping model. The preset training termination condition includes loss convergence and reaching a maximum iteration threshold, etc.

[0096] The embodiments of the present invention, through the above-described scheme, enable the initial mapping model to be trained based on structured log text and the log template to be trained. This helps to control the loss value of the initial mapping model within a preset range, thereby enabling the trained initial mapping model to map log text to log templates, ensuring the uniqueness of log text classification, avoiding ambiguity, and improving the accuracy of log anomaly detection.

[0097] Figure 3 This is the third flowchart illustrating the log anomaly detection method provided by this invention. For example... Figure 3 As shown, in one embodiment of the present invention, after performing clustering processing on each of the structured log texts to obtain each log template to be trained, the method further includes:

[0098] Step 31: Perform string conversion processing on each of the log templates to be trained to obtain the hash value corresponding to each log template to be trained;

[0099] It should be noted that string conversion refers to the process of converting the log templates to be trained into strings. This process includes methods such as MD2, MD2, and MD5; MD5 is selected. Specifically, by performing MD5 conversion on each of the log templates to be trained, the corresponding hash value can be obtained. By calculating the MD5 hash of the log templates to be trained, the MD5 hash value is used as a marker for log classification. Due to the characteristics of MD5, a one-to-one correspondence between the log templates to be trained and the MD5 hash value can be ensured.

[0100] Step 32: Configure the log tag information corresponding to each hash value;

[0101] It should be noted that since the MD5 hash value is a string of length 32, the computational workload in the subsequent data processing of the MD5 hash value may be relatively complex. Therefore, in this embodiment, each hash value is configured with log tag information of a shorter data type. Preferably, the log tag information is an int integer value.

[0102] Specifically, each of the log templates to be trained and each of the hash values ​​are associated and stored in a preset database. During the storage process, corresponding label information, namely, log label information in this embodiment, is configured or generated for each hash value, thereby determining the correspondence between the hash value and the log label information.

[0103] Step 33: Construct a mapping relationship between each of the log templates to be trained and each of the log tag information, wherein the mapping relationship is used to map the mapped log templates to the target log tag information.

[0104] Specifically, based on the one-to-one correspondence between the log template to be trained and the MD5 hash value, and the correspondence between the hash value and the log tag information, a mapping relationship between each of the log templates to be trained and each of the log tag information is constructed. Thus, based on the mapping relationship, the log template can be mapped to obtain the log tag information corresponding to the log template.

[0105] Alternatively, in other embodiments, ID classification labels can be directly configured for the log template to be trained, and the ID classification labels can be used as the target log label information.

[0106] This invention improves computational efficiency by performing MD5 calculation on log templates and using the MD5 hash value as a log classification identifier. This ensures a one-to-one correspondence between log templates and MD5 hash values, and converts the MD5 hash value into log tag information with a shorter data type.

[0107] Figure 4 This is the fourth flowchart illustrating the log anomaly detection method provided by this invention. Figure 4 As shown, in one embodiment of the present invention, the log anomaly detection module is trained based on the following steps:

[0108] Step 41: Group the structured log texts to obtain several sets of historical log sequences;

[0109] It should be noted that the grouping process is based on the system's business logic. Specifically, the business logic of the target system is determined, and then, according to the business logic, the structured log texts are grouped to obtain several sets of historical log sequences.

[0110] Step 42: Based on the mapping relationship, the training log templates corresponding to each structured log text in each historical log sequence are mapped to obtain the mapping label information corresponding to each historical log sequence.

[0111] Step 43: Based on the mapping label information corresponding to each of the historical log sequences, form each training flag sequence;

[0112] Specifically, the following steps are performed for any set of historical log sequences:

[0113] By mapping the training log templates corresponding to each structured log text in the historical log sequence according to the mapping relationship, the mapping label information associated with each training log template can be obtained. Further, based on the mapping label information corresponding to each historical log sequence, a training flag sequence corresponding to the historical log sequence is formed. Understandably, the grouped structured log texts are replaced using the mapping label information.

[0114] Step 44: Based on each of the training flag sequences, iteratively train the training detection model to obtain the log detection model.

[0115] Step 44 above, which involves iteratively training the detection model to obtain the log detection model based on each of the training target sequence, includes: for any training target sequence, using the sliding window to truncate the mapping label information in the training target sequence to obtain a sequence subset corresponding to each sliding window; taking each sequence subset corresponding to the sliding window as a training sample, and taking the next mapping label information corresponding to the sliding window as the sample label of the training sample; and iteratively training the detection model to obtain the log detection model based on multiple training samples and their respective sample labels.

[0116] It should be noted that the lengths of the various training flag sequences are not the same. In addition, the log detection models include time series models such as RNN, LSTM and Transformer. In this embodiment, in order to improve data processing efficiency, the detection model to be trained is the Transformer model.

[0117] Specifically, the following steps are performed for any given sequence of flags to be trained:

[0118] The training flag sequence is truncated by sliding a window of a pre-set length to obtain a sequence subset corresponding to each sliding window. For example, if the training flag sequence is [1, 2, 3, ..., 8, 9, 10, 11, 12] and the length of the sliding window is 10, then the data of [1, 2, 3, ..., 8, 9, 10] is taken as the sequence subset corresponding to the first sliding window, the data of [2, 3, ..., 8, 9, 10, 11] is taken as the sequence subset corresponding to the second sliding window, and so on.

[0119] Furthermore, for any sliding window, the sequence subset corresponding to the sliding window is taken as a training sample, and the next mapping label information corresponding to the sliding window is taken as the sample label of the training sample. Following the above example, the data of [1, 2, 3, ..., 8, 9, 10] is taken as the sequence subset corresponding to the first sliding window, and the next mapping label information corresponding to the sliding window is represented as

[11] . Furthermore, after obtaining multiple training samples, based on the multiple training samples and the detection model to be trained, a predicted value is obtained. According to the predicted value and the sample label corresponding to the training sample, the model parameters in the detection model to be trained are adjusted to obtain the log detection model.

[0120] In other embodiments, based on the training log template corresponding to each structured log text in each historical log sequence, the hash values ​​corresponding to any historical log sequence are retrieved from a preset database, thereby forming multiple hash value sequences. A log detection model is then trained based on these hash value sequences. In this case, during the detection application process, the sequences in the hash value formation process are also input into the log detection model for detection.

[0121] This invention grouped structured log texts according to business logic and transformed each group of structured log texts into a training label sequence formed by mapping label information. Then, iteratively trained the log detection model based on the training label sequence, which can effectively improve the training effect of the model. Furthermore, the Transformer model can process data in parallel, thereby accelerating the training and inference time of the fast and efficient anomaly detection process.

[0122] The log anomaly detection device provided by the present invention is described below. The log anomaly detection device described below can be referred to in correspondence with the log anomaly detection method described above.

[0123] Figure 5 This is a schematic diagram of the log anomaly detection device provided by the present invention, as shown below. Figure 5 As shown, an embodiment of the present invention provides a log anomaly detection device, the device comprising:

[0124] Module 51 is used to acquire several target log texts corresponding to the target system;

[0125] The mapping module 52 is used to input each of the target log texts into the log template mapping model to obtain the mapped log templates corresponding to each target log text output by the log template mapping model.

[0126] The forming module 53 is used to map each of the mapped log templates to target log tag information to form multiple log tag sequences;

[0127] The detection module 54 is used to slide and extract any of the log tag sequences based on a pre-set sliding window, so as to input the log tag subset corresponding to any sliding window into the log detection model and obtain the detection result corresponding to any sliding window output by the log detection model.

[0128] The determining module 55 is used to determine the final log detection result based on the detection result corresponding to any sliding window and the next target log tag information corresponding to the sliding window;

[0129] The log template mapping model is obtained through iterative training based on each structured log text and the corresponding log template to be trained for each structured log text; the log template to be trained is obtained by clustering the structured log text.

[0130] The log detection model is obtained by iterative training based on each training flag sequence; the training flag sequence is obtained by mapping each training log template.

[0131] The log anomaly detection device includes:

[0132] Obtain the log codes generated during the operation of the target system;

[0133] Based on the log codes described above, several original log texts are selected from the historical log files;

[0134] The original log texts are processed into structured log texts to obtain structured log texts.

[0135] Clustering is performed on each of the structured log texts to obtain each log template to be trained;

[0136] Based on the structured log texts and the log templates to be trained, the initial mapping model is iteratively trained to obtain the log template mapping model.

[0137] The log anomaly detection device includes:

[0138] Each of the log templates to be trained is processed by string conversion to obtain the hash value corresponding to each log template to be trained.

[0139] Configure the log tag information corresponding to each hash value;

[0140] Construct a mapping relationship between each of the log templates to be trained and each of the log tag information, wherein the mapping relationship is used to map the mapped log templates to the target log tag information.

[0141] The log anomaly detection device includes:

[0142] The structured log texts are grouped to obtain several sets of historical log sequences;

[0143] Based on the mapping relationship, the training log templates corresponding to each structured log text in each of the historical log sequences are mapped to obtain the mapping label information corresponding to each of the historical log sequences.

[0144] Based on the mapping label information corresponding to each of the historical log sequences, each training flag sequence is formed;

[0145] Based on the aforementioned training flag sequences, the training detection model is iteratively trained to obtain the log detection model.

[0146] The log anomaly detection device includes:

[0147] Each of the sequence subsets corresponding to the sliding window is used as a training sample, and the next mapping label information corresponding to the sliding window is used as the sample label of the training sample.

[0148] Based on multiple training samples and their corresponding sample labels, the detection model to be trained is iteratively trained to obtain the log detection model.

[0149] The forming module 53 is further configured to:

[0150] The target log texts are grouped to obtain log group sequences;

[0151] According to the mapping relationship, the mapping log template corresponding to each target log text in any log grouping sequence is mapped to the target log tag information;

[0152] Each log tag sequence is formed based on the target log tag information corresponding to any of the log grouping sequences.

[0153] The determining module 55 is further configured to:

[0154] The detection result is matched with the next target log tag information in the sliding window;

[0155] If the match is successful, the final log detection result is determined to be in a normal state;

[0156] If the match fails, the final log detection result is determined to be an abnormal state.

[0157] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail here.

[0158] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a memory 620, a communications interface 630, and a communications bus 640. The processor 610, memory 620, and communications interface 630 communicate with each other via the communications bus 640. The processor 610 can call logical instructions from the memory 620 to execute a log anomaly detection method.

[0159] Furthermore, the logical instructions in the aforementioned memory 620 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the log anomaly detection methods provided by the methods described above.

[0161] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the log anomaly detection method provided by the above methods.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / content> < / func> < / task> < / container> < / level> < / type> < / time> < / date>

Claims

1. A method for detecting log anomalies, characterized in that, include: Obtain several target log texts corresponding to the target system; Each of the target log texts is input into the log template mapping model to obtain the mapped log template corresponding to each target log text output by the log template mapping model; the log template mapping model is used to ensure that the mapped log template corresponding to the same log text is uniquely determined; Each of the mapped log templates is mapped to target log tag information, forming multiple log tag sequences; Based on a pre-set sliding window, any of the log tag sequences is slid-trimmed to input the log tag subset corresponding to any sliding window into the log detection model, thereby obtaining the detection result corresponding to any sliding window output by the log detection model. For any detection result corresponding to a sliding window, the final log detection result is determined based on the detection result and the next target log tag information corresponding to the sliding window; The log template mapping model is obtained through iterative training based on each structured log text and the corresponding log template to be trained for each structured log text; the log template to be trained is obtained by clustering the structured log text. The log detection model is obtained by iterative training based on each training flag sequence; the training flag sequence is obtained by mapping each training log template. The log template mapping model is trained based on the following steps: Obtain the runtime code file of the target system, search for multiple log output packets in the runtime code file, search for multiple output log codes based on each log output packet, and use the multiple output log codes as multiple log codes; In the historical log files, log texts that cover all the log codes are selected. The similarity between each log text and the log code is calculated using a preset similarity calculation algorithm. Log texts with a preset number of texts that have a high degree of similarity are selected as several original log texts. The preset similarity calculation algorithm is the TF-IDF algorithm. The original log texts are processed into structured log texts to obtain structured log texts. Clustering is performed on each of the structured log texts to obtain each log template to be trained; Based on the structured log texts and the log templates to be trained, the initial mapping model is iteratively trained to obtain the log template mapping model. After performing clustering processing on each of the structured log texts to obtain each log template to be trained, the method further includes: Each of the log templates to be trained is processed by string conversion to obtain the hash value corresponding to each log template to be trained. Each of the log templates to be trained and each of the hash values ​​are associated and stored in a preset database, and corresponding log tag information is configured for each of the hash values; the log tag information is an integer value. Based on the correspondence between each of the log templates to be trained and each of the hash values, and the correspondence between each of the hash values ​​and each of the log tag information, a mapping relationship between each of the log templates to be trained and each of the log tag information is constructed, wherein the mapping relationship is used to map the mapped log template to the target log tag information; The step of mapping each of the mapped log templates to target log tag information to form multiple log tag sequences includes: According to the business logic of the target system, the target log texts are grouped to obtain various log grouping sequences; when the microservice is a distributed architecture, it is grouped in the following order: cluster-node-container-microservice-function-thread; when performing log detection on a single microservice, it is grouped according to the hierarchical structure of container-microservice-function-thread. According to the mapping relationship, the mapping log template corresponding to each target log text in any log grouping sequence is mapped to the target log tag information; Each log tag sequence is formed based on the target log tag information corresponding to any of the log grouping sequences.

2. The log anomaly detection method according to claim 1, characterized in that, The log anomaly detection module is trained based on the following steps: The structured log texts are grouped to obtain several sets of historical log sequences; Based on the mapping relationship, the training log templates corresponding to each structured log text in each of the historical log sequences are mapped to obtain the mapping label information corresponding to each of the historical log sequences. Based on the mapping label information corresponding to each of the historical log sequences, each training flag sequence is formed; Based on the aforementioned training flag sequences, the detection model is iteratively trained to obtain the log detection model.

3. The log anomaly detection method according to claim 2, characterized in that, The step of iteratively training the detection model based on each of the training flag sequences to obtain the log detection model includes: For any training label sequence, the sliding window is used to slide and extract the mapping label information in the training label sequence to obtain a sequence subset corresponding to each sliding window; Each of the sequence subsets corresponding to the sliding window is used as a training sample, and the next mapping label information corresponding to the sliding window is used as the sample label of the training sample. Based on multiple training samples and their corresponding sample labels, the detection model to be trained is iteratively trained to obtain the log detection model.

4. The log anomaly detection method according to claim 1, characterized in that, For any sliding window, the final log detection result is determined based on the detection result and the next target log tag information corresponding to the sliding window, including: The detection result is matched with the next target log tag information in the sliding window; If the match is successful, the final log detection result is determined to be in a normal state; If the match fails, the final log detection result is determined to be an abnormal state.

5. A log anomaly detection device, characterized in that, include: The acquisition module is used to acquire several target log texts corresponding to the target system; The mapping module is used to input each of the target log texts into the log template mapping model to obtain the mapped log templates corresponding to each target log text output by the log template mapping model; the log template mapping model is used to ensure that the mapped log template corresponding to the same log text is uniquely determined; A forming module is used to map each of the mapped log templates to target log tag information, thereby forming multiple log tag sequences; The detection module is used to slide and truncate any of the log tag sequences based on a pre-set sliding window, so as to input the log tag subset corresponding to any sliding window into the log detection model and obtain the detection result corresponding to any sliding window output by the log detection model. The determination module is used to determine the final log detection result based on the detection result corresponding to any sliding window and the next target log tag information corresponding to the sliding window. The log template mapping model is obtained through iterative training based on each structured log text and the corresponding log template to be trained for each structured log text; the log template to be trained is obtained by clustering the structured log text. The log detection model is obtained by iterative training based on each training flag sequence; the training flag sequence is obtained by mapping each training log template. The log template mapping model is trained based on the following steps: Obtain the runtime code file of the target system, search for multiple log output packets in the runtime code file, search for multiple output log codes based on each log output packet, and use the multiple output log codes as multiple log codes; In the historical log files, log texts that cover all the log codes are selected. The similarity between each log text and the log code is calculated using a preset similarity calculation algorithm. Log texts with a preset number of texts that have a high degree of similarity are selected as several original log texts. The preset similarity calculation algorithm is the TF-IDF algorithm. The original log texts are processed into structured log texts to obtain structured log texts. Clustering is performed on each of the structured log texts to obtain each log template to be trained; Based on the structured log texts and the log templates to be trained, the initial mapping model is iteratively trained to obtain the log template mapping model. After performing clustering processing on each of the structured log texts to obtain each log template to be trained, the method further includes: Each of the log templates to be trained is processed by string conversion to obtain the hash value corresponding to each log template to be trained. Each of the log templates to be trained and each of the hash values ​​are associated and stored in a preset database, and corresponding log tag information is configured for each of the hash values; the log tag information is an integer value. Based on the correspondence between each of the log templates to be trained and each of the hash values, and the correspondence between each of the hash values ​​and each of the log tag information, a mapping relationship between each of the log templates to be trained and each of the log tag information is constructed, wherein the mapping relationship is used to map the mapped log template to the target log tag information; The step of mapping each of the mapped log templates to target log tag information to form multiple log tag sequences includes: According to the business logic of the target system, the target log texts are grouped to obtain various log grouping sequences; when the microservice is a distributed architecture, it is grouped in the following order: cluster-node-container-microservice-function-thread; when performing log detection on a single microservice, it is grouped according to the hierarchical structure of container-microservice-function-thread. According to the mapping relationship, the mapping log template corresponding to each target log text in any log grouping sequence is mapped to the target log tag information; Each log tag sequence is formed based on the target log tag information corresponding to any of the log grouping sequences.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the log anomaly detection method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the log anomaly detection method as described in any one of claims 1 to 4.

Images