Log anomaly detection method and device based on deep learning, equipment and medium

Abstract

The invention discloses a log anomaly detection method and device based on deep learning, equipment and a medium. The method comprises the following steps: firstly, obtaining a real-time static template and a real-time dynamic parameter according to a real-time log and an n-gram template; vectorizing the real-time static template and the real-time dynamic parameter to obtain a real-time semantic vector and a real-time dynamic parameter vector which represent a specific log template; synthesizing the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector; performing context analysis on the real-time dynamic event vector by using a log anomaly detection model to obtain an expression rule of a real-time log; and if the performance rule of the real-time log is consistent with the performance rule of the abnormal log, detecting that the real-time log is abnormal. In the process, context analysis is carried out on the real-time log by utilizing the log anomaly detection model to obtain a performance rule, and whether the current real-time log belongs to the abnormal log or not is detected according to the performance rule, so that the anomaly detection accuracy and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the technical field of log anomaly detection, and in particular to a log anomaly detection method, device, equipment and medium based on deep learning. Background Art

[0002] With the rapid development of Internet technology, more and more businesses use cutting-edge Internet computer technologies to improve business processing efficiency. In order to ensure the efficiency and security of business processing, certain requirements are required for the availability and stability of the system. The system operation log can always record the dynamic information of the system operation and reflect the operating status of the system at all times. This has important practical significance for system monitoring, early warning and fault handling. Log-based anomaly detection technology is to timely discover and remind the system of abnormal behavior and status.

[0003] Currently, the log-based anomaly detection technology is a method of manually downloading logs for manual analysis. However, the business system has a large amount of data, especially the intelligent terminal system has a large number of users and transactions, and the types and numbers of hardware modules involved are large. The number of logs generated is extremely large and complex, which leads to low efficiency of manual processing and is prone to errors, resulting in low accuracy. Summary of the invention

[0004] The present application provides a log anomaly detection method, apparatus, device and medium based on deep learning, which adopts deep learning technology to analyze and obtain high-dimensional features from log data, thereby improving the efficiency and accuracy of the anomaly detection method.

[0005] In a first aspect, the present application provides a log anomaly detection method based on deep learning, the method comprising:

[0006] According to the real-time log and the n-gram template, a real-time static template and a real-time dynamic parameter are obtained, wherein the n-gram template is a statistical language model with a tree structure of a fixed depth;

[0007] Vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template, and vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector;

[0008] The real-time semantic vector and the real-time dynamic parameter vector are synthesized into a real-time dynamic event vector;

[0009] Use a log anomaly detection model to perform context analysis on the real-time dynamic event vector to obtain the performance rules of the real-time log. The log anomaly detection model is a trained Bi-LSTM model based on the attention mechanism. The training samples of the log anomaly detection model include the performance rules of historical logs and the historical dynamic event vectors of historical logs.

[0010] If the performance pattern of the real-time log is consistent with the performance pattern of the abnormal log, the real-time log is detected to be abnormal.

[0011] Optionally, according to the real-time log and the n-gram template, a real-time static template and a real-time dynamic parameter are obtained, including:

[0012] Preprocess the real-time log to obtain the real-time log statement;

[0013] The real-time log statements are input into the n-gram template for matching, and the real-time static templates matching the n-gram templates and the real-time dynamic parameters that cannot match the n-gram templates are obtained.

[0014] Optionally, the process of constructing an n-gram template includes:

[0015] Preprocess the historical logs to obtain historical log statements;

[0016] Extract the frequently occurring byte fragment sequence of length N from the historical log statements;

[0017] According to the byte segment sequence of length N, a historical static template dictionary is constructed;

[0018] Construct n-gram templates based on the historical static template dictionary.

[0019] Optionally, the real-time static template is vectorized to obtain a real-time semantic vector representing a specific log template, including:

[0020] Preprocess the real-time static template and delete the non-character part of the real-time static template;

[0021] Use the TF-IDF method to calculate the weight value of the word vector in the preprocessed real-time static template;

[0022] The word vectors in the preprocessed real-time static template are weighted and added to the corresponding weight values ​​to obtain a real-time semantic vector representing a specific log template.

[0023] Optionally, the real-time dynamic parameter is vectorized to obtain a real-time dynamic parameter vector, including:

[0024] Use fully connected neural networks to obtain hidden information of real-time dynamic parameters;

[0025] The softmax function is used to calculate the hidden information of the real-time dynamic parameters, the initialized weight vector and the bias vector to obtain the real-time dynamic parameter vector.

[0026] Optionally, the training process of the log anomaly detection model includes:

[0027] Process historical logs to obtain historical static templates and historical dynamic parameters;

[0028] Vectorizing the historical static template and the historical dynamic parameter respectively to obtain the historical semantic vector and the historical dynamic parameter vector;

[0029] The historical semantic vector and the historical dynamic parameter vector are synthesized into a historical dynamic event vector;

[0030] The performance rules of historical logs and historical dynamic event vectors are used as training samples and input into the Bi-LSTM model based on the attention mechanism for training;

[0031] When the Bi-LSTM model based on the attention mechanism meets the training conditions, a log anomaly detection model is obtained.

[0032] Optionally, after detecting that the real-time log is abnormal, the following steps are further included:

[0033] Manually identify the real-time logs and obtain identification results;

[0034] If the judgment result is normal, the parameters of the log anomaly detection model are updated;

[0035] If the judgment result is abnormal, the corresponding treatment will be adopted according to the abnormal problem.

[0036] In a second aspect, the present application provides a log anomaly detection device based on deep learning, the device comprising:

[0037] An obtaining unit, used for obtaining a real-time static template and a real-time dynamic parameter according to a real-time log and an n-gram template, wherein the n-gram template is a statistical language model with a tree structure of a fixed depth;

[0038] A vectorization unit, used for vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template, and vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector;

[0039] A synthesis unit, used for synthesizing the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector;

[0040] An analysis unit, configured to perform context analysis on the real-time dynamic event vector using a log anomaly detection model to obtain a performance rule of the real-time log, wherein the log anomaly detection model is a trained Bi-LSTM model based on an attention mechanism, and the training samples of the log anomaly detection model include performance rules of historical logs and historical dynamic event vectors of historical logs;

[0041] The detection unit is used to detect that an abnormality occurs in the real-time log if a performance rule of the real-time log is consistent with a performance rule of the abnormal log.

[0042] Optionally, the obtaining unit is specifically used for:

[0043] Preprocess the real-time log to obtain the real-time log statement;

[0044] The real-time log statements are input into the n-gram template for matching, and the real-time static templates matching the n-gram templates and the real-time dynamic parameters that cannot match the n-gram templates are obtained.

[0045] Optionally, the device further comprises a construction unit for:

[0046] Preprocess the historical logs to obtain historical log statements;

[0047] Extract the frequently occurring byte fragment sequence of length N from the historical log statements;

[0048] According to the byte segment sequence of length N, a historical static template dictionary is constructed;

[0049] Construct n-gram templates based on the historical static template dictionary.

[0050] Optionally, the vectorization unit vectorizes the real-time static template to obtain a real-time semantic vector representing a specific log template, which is specifically used for:

[0051] Preprocess the real-time static template and delete the non-character part of the real-time static template;

[0052] Use the TF-IDF method to calculate the weight value of the word vector in the preprocessed real-time static template;

[0053] The word vectors in the preprocessed real-time static template are weighted and added to the corresponding weight values ​​to obtain a real-time semantic vector representing a specific log template.

[0054] Optionally, the vectorization unit vectorizes the real-time dynamic parameter to obtain a real-time dynamic parameter vector, which is specifically used for:

[0055] Use fully connected neural networks to obtain hidden information of real-time dynamic parameters;

[0056] The softmax function is used to calculate the hidden information of the real-time dynamic parameters, the initialized weight vector and the bias vector to obtain the real-time dynamic parameter vector.

[0057] Optionally, the device further comprises a training unit for:

[0058] Process historical logs to obtain historical static templates and historical dynamic parameters;

[0059] Vectorize the historical static template and the historical dynamic parameters respectively to obtain the historical semantic vector and the historical dynamic parameter vector;

[0060] The historical semantic vector and the historical dynamic parameter vector are synthesized into a historical dynamic event vector;

[0061] The performance rules of historical logs and historical dynamic event vectors are used as training samples and input into the Bi-LSTM model based on the attention mechanism for training;

[0062] When the Bi-LSTM model based on the attention mechanism meets the training conditions, a log anomaly detection model is obtained.

[0063] Optionally, after detecting that the real-time log is abnormal, the device further includes:

[0064] A determination unit, used for manually determining the real-time log to obtain a determination result;

[0065] The processing unit is used to update the parameters of the log anomaly detection model if the judgment result is normal; if the judgment result is abnormal, adopt a response process according to the abnormal problem.

[0066] In a third aspect, the present application provides an electronic device, the electronic device comprising a memory and a processor:

[0067] The memory is used to store computer programs;

[0068] The processor is used to execute the method provided by the first aspect according to the computer program.

[0069] In a fourth aspect, the present application further provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method provided in the first aspect above.

[0070] It can be seen that this application has the following beneficial effects:

[0071] The present application provides a log anomaly detection method based on deep learning. First, according to the real-time log and n-gram template, a real-time static template and a real-time dynamic parameter are obtained. The n-gram template is a statistical language model with a tree structure of fixed depth; the real-time static template is vectorized to obtain a real-time semantic vector representing a specific log template, and the real-time dynamic parameter is vectorized to obtain a real-time dynamic parameter vector; the real-time semantic vector and the real-time dynamic parameter vector are synthesized into a real-time dynamic event vector; the real-time dynamic event vector is contextually analyzed using a log anomaly detection model to obtain the performance law of the real-time log, the log anomaly detection model is a trained Bi-LSTM model based on an attention mechanism, and the training samples of the log anomaly detection model include the performance law of the historical log and the historical dynamic event vector of the historical log; if the performance law of the real-time log is consistent with the performance law of the abnormal log, the real-time log is detected to be abnormal. In this process, the performance law of the real-time log can be obtained by using the log anomaly detection model to perform contextual analysis on the real-time log, and whether the current real-time log belongs to the abnormal log is detected according to the performance law, thereby improving the accuracy and efficiency of anomaly detection. In addition, the real-time static template and real-time dynamic parameters of the input real-time log are decomposed in this application, so that the purpose of online log analysis can be achieved without waiting for log batch collection, which lays the foundation for the system to detect anomalies in real time and respond quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0073] Figure 1 This is a flow chart of an embodiment of a method for detecting anomalies in a log based on deep learning in an embodiment of the present application;

[0074] Figure 2 This is a schematic diagram of a LogParser tool architecture in an embodiment of the present application;

[0075] Figure 3 This is a schematic diagram of the architecture of a log anomaly detection framework LogDection based on deep learning in an embodiment of the present application;

[0076] Figure 4 This is a flow chart of another embodiment of a log anomaly detection method based on deep learning in the embodiments of the present application;

[0077] Figure 5This is a schematic diagram of the structure of the Bi-LSTM model based on the attention mechanism in the embodiment of the present application;

[0078] Figure 6 This is a structural diagram of a log anomaly detection device based on deep learning in an embodiment of the present application;

[0079] Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0082] First, the technical terms appearing in the embodiments of the present application are explained;

[0083] (1) Intelligent terminal equipment: "Intelligent terminal equipment" is the business acceptance equipment at the branch end of the personal business centralized operation system. Through the integration of hardware equipment and software systems, it can handle the business requests of most individual customers.

[0084] (2) N-Gram: N-Gram is an algorithm based on a statistical language model. It performs a sliding window operation of size N on the text bytes to form a sequence of byte segments of length N, each of which is called a gram.

[0085] (3) Embedding vector: An embedding vector is a representation of an entity, and can also be seen as a set of features extracted from the entity. The samples (sentences, articles) in natural language processing-related problems are a bunch of words (Tokens such as n-grams). The words, people, objects and other entities in the samples can be converted into embedding vectors first, and then sent to the subsequent part of the neural network, so that the machine learning model can effectively process and analyze them.

[0086] (4) TF-IDF clustering algorithm: TF-IDF is a statistical method used to evaluate the importance of a document set or a document in a corpus. The importance of a word is proportional to the number of times it appears, but it is inversely proportional to the frequency of its appearance in the corpus.

[0087] At present, intelligent terminal devices are an organic combination of Internet technology and the financial industry. With the support of Internet technology, the hardware equipment of the financial industry integrates various services including biometric technology and financial big data cloud platform, which can quickly process complex business, support multiple payment methods and multiple languages, improve the processing efficiency of various banking businesses, improve transaction security, improve customer experience and reduce the pressure of branch counters. In recent years, cloud computing technology has been increasingly popularized, and cloud platform security has become increasingly important. Cloud platforms have characteristics such as distribution, and data is vulnerable to attacks. Inserting log print statements in the code can generate a large number of logs when the system is running, including the current operating status, detailed information of errors, user operation processes, etc. These log information is an important basis for anomaly detection.

[0088] However, due to the large number of transactions and the large transaction volume of the intelligent terminal system, the logs generated by the intelligent terminal system are also huge, which makes it extremely difficult to collect and analyze the logs. At present, operation and maintenance personnel usually check the system logs by manually searching for keywords to detect anomalies, but the manual detection method is inefficient and prone to errors, resulting in low accuracy.

[0089] In the embodiments of the present application, a log anomaly detection method based on deep learning is used, which uses deep learning technology to analyze and obtain high-dimensional features from log data, thereby greatly improving the efficiency and accuracy of the anomaly detection method.

[0090] In specific implementation, the method may include, for example: first, obtaining a real-time static template and a real-time dynamic parameter according to a real-time log and an n-gram template, wherein the n-gram template is a statistical language model using a tree structure with a fixed depth; vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template, and vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector; synthesizing the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector; using a log anomaly detection model to perform context analysis on the real-time dynamic event vector to obtain a performance rule of the real-time log, wherein the log anomaly detection model is a trained Bi-LSTM model based on an attention mechanism, and the training samples of the log anomaly detection model include a performance rule of a historical log and a historical dynamic event vector of the historical log; if the performance rule of the real-time log is consistent with the performance rule of the abnormal log, then detecting that the real-time log is abnormal.

[0091] It can be seen that the method provided by the implementation of this application can obtain the performance rules of real-time logs by using the log anomaly detection model to perform context analysis on real-time logs, and detect whether the current real-time log is an abnormal log based on the performance rules, thereby improving the accuracy and efficiency of anomaly detection. In addition, the real-time static template and real-time dynamic parameters of the input real-time log are decomposed in this application, and the purpose of online log parsing can be achieved without waiting for the log batch collection, which lays the foundation for the system to detect anomalies in real time and respond quickly.

[0092] To facilitate understanding of the specific implementation of the log anomaly detection method based on deep learning provided in the embodiment of the present application, the following will be described in conjunction with the accompanying drawings.

[0093] It should be noted that the subject implementing the deep learning-based log anomaly detection method may be a deep learning-based log anomaly detection device provided in the embodiment of the present application, and the deep learning-based log anomaly detection device may be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiment of the present application may be any device capable of implementing the deep learning-based log anomaly detection method in the embodiment of the present application, such as an Internet of Things (IoT) device.

[0094] Figure 1 A flow chart of a log anomaly detection method based on deep learning provided in an embodiment of the present application. The method can be applied to a log anomaly detection device based on deep learning, which can be, for example, Figure 6 The log anomaly detection device 600 based on deep learning is shown, or the log anomaly detection device based on deep learning can also be integrated in Figure 7 Functional modules in the electronic device 700 are shown.

[0095] like Figure 1 As shown, the method includes the following S101 to S105:

[0096] S101: obtaining a real-time static template and real-time dynamic parameters according to a real-time log and an n-gram template, wherein the n-gram template is a statistical language model with a tree structure of a fixed depth.

[0097] In order to realize log anomaly detection, the embodiment of the present application first obtains the real-time static template and the real-time dynamic parameter according to the real-time log and the n-gram template; vectorizes the real-time static template to obtain the real-time semantic vector representing the specific log template, and vectorizes the real-time dynamic parameter to obtain the real-time dynamic parameter vector; synthesizes the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector; then uses the log anomaly detection model to perform context analysis on the real-time dynamic event vector to obtain the performance law of the real-time log, and finally, if the performance law of the real-time log is consistent with the performance law of the abnormal log, the real-time log is detected to be abnormal. Therefore, the embodiment of the present application obtains the real-time static template and the real-time dynamic parameter through S101 to make preliminary preparations for the subsequent acquisition of the real-time semantic vector and the real-time dynamic parameter vector.

[0098] First, the construction process of n-gram template is described, such as Figure 2 As shown in the LogParser tool architecture diagram, before building the n-gram template, it is necessary to build the log dictionary LogDic. The specific process of building the log dictionary LogDic may include: judging from the operation and maintenance experience, the n-gram words that appear more frequently in the log belong to the static template, while the n-gram words with lower frequency belong to the dynamic variable. Therefore, the historical logs are first preprocessed to obtain the historical log statements; the byte fragment sequences with a length of N that appear frequently are extracted from the historical log statements; and the historical static template dictionary LogDic is constructed based on the byte fragment sequence with a length of N. Thus, the n-gram template is constructed based on the historical static template dictionary.

[0099] After obtaining the n-gram template, the real-time log can be matched, such as Figure 2 As shown, as an example, S101 may include: preprocessing the real-time log to obtain the real-time log statement; inputting the real-time log statement into the n-gram template for matching, obtaining the real-time static template that matches the n-gram template and the real-time dynamic parameters that cannot match the n-gram template.

[0100] Among them, since the amount of data in the LogDict dictionary is very large, traversing the dictionary is very time-consuming and cannot meet the parsing requirements. Therefore, the LogParser tool in the embodiment of the present application uses a fixed-depth tree structure for matching. The child nodes of the fixed-depth parsing tree are only 2-gram and 3-gram nodes, and the parsing efficiency will not decrease due to the large amount of data. The matching algorithm is shown in Table 1 below.

[0101]

[0102]

[0103] In the above Table 1, the input is the real-time log, and the output is the real-time static template LogList that matches the n-gram template. The process is to first establish a list LogList (real-time static template) that stores the LOGMap that matches the n-gram template, and then pre-process the real-time log to obtain the word unit tokens list S (real-time log statement). When the list S is not empty, query the byte segments of the length corresponding to 2-gram and 3-gram, and query the word units in the corresponding byte segments, and then query the word units that match the log dictionary LogDic in the n-gram template and store them in LOGMap. Finally, if LOGMap is not empty, add LOGMap to LogList to obtain the real-time static template.

[0104] It should be noted that the child nodes of the fixed depth parsing tree in the embodiment of the present application not only indicate the two nodes of 2-gram and 3-gram, but can also be other nodes, which is not limited here.

[0105] In this process, the log parsing tool LogParser can be used to decompose the static template and dynamic parameters of the input log stream in real time from the device side. The purpose of online log parsing can be achieved without waiting for log batch collection, laying the foundation for the system to detect anomalies in real time and respond quickly.

[0106] S102: vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template, and vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector.

[0107] As an example, the process of obtaining a real-time semantic vector representing a specific log template in S102 may include: preprocessing the real-time static template and deleting non-character parts in the real-time static template; using the TF-IDF method to calculate the weight value of the word vector in the preprocessed real-time static template; weighted addition of the word vector in the preprocessed real-time static template and the corresponding weight value to obtain a real-time semantic vector representing the specific log template.

[0108] Since the smart terminal system will be continuously upgraded, the logs will change. Single static template matching cannot adapt to such changes. Therefore, it is necessary to convert the static template into an embedding vector and map the static template into a semantic vector of fixed dimension. By calculating the distance between the semantic vectors, the iteration degree of the log can be analyzed.

[0109] The above preprocesses the real-time static template, that is, some non-character word units in the log will interfere with the extraction of semantic information and need to be deleted from the real-time static template.

[0110] The above TF-IDF clustering process clusters all word vectors and can represent the real-time static template as a semantic vector of fixed dimension. Therefore, when the number of word vectors in the real-time static template changes, the dimension of its semantic vector remains unchanged. The weight value is calculated using the TF-IDF method for each word in the static template. Finally, all word vectors are multiplied and accumulated with the corresponding TF-IDF weight value to obtain the real-time semantic vector representing the specific log template. The formula is as follows:

[0111]

[0112] Among them, E sem represents the real-time semantic vector; N represents the length of the real-time static template; q i represents the i-th weight value; v i Represents the i-th word vector.

[0113] The semantic information extraction of the above static template obtains the system execution path exception by analyzing the order of event sequences.

[0114] System performance anomalies will manifest as disordered changes in system parameters, and disordered parameters will be reflected in their vectorized form, so it is necessary to vectorize log dynamic parameters.

[0115] As an example, the process of obtaining the real-time dynamic parameter vector in S102 may include: using a fully connected neural network to obtain hidden information of the real-time dynamic parameters; using a softmax function to calculate the hidden information of the real-time dynamic parameters, the initialized weight vector and the bias vector to obtain the real-time dynamic parameter vector.

[0116] The real-time dynamic parameters of each log record above form a one-dimensional parameter value list C = [C 1 ,C 2 ,…,C M ]. LogDection uses a fully connected neural network to obtain the hidden information of real-time dynamic parameters and project the one-dimensional vector into a dimensional space. The vectorization process is to fill it with a fixed-dimensional one-dimensional vector, multiply it by a randomly initialized weight vector, add a bias vector, and finally apply the softmax function to map it to obtain the real-time dynamic parameter vector. The formula is as follows:

[0117] E par =softmax(W T C+b),W∈R d ,b∈R d

[0118] Among them, E par represents the real-time dynamic parameter vector; W represents the weight vector; C represents the one-dimensional parameter value list; b represents the bias vector.

[0119] S103: Synthesize the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector.

[0120] After the above process, a single log can be expressed as a real-time semantic vector E sem and a dynamic parameter vector E par , which requires log anomaly detection, requiring E sem With E par The connection can obtain a dynamic event vector, which is a real-time dynamic event vector synthesized by the real-time semantic vector and the real-time dynamic parameter vector, for subsequent log anomaly detection.

[0121] S104: Use a log anomaly detection model to perform context analysis on the real-time dynamic event vector to obtain the performance rules of the real-time log, wherein the log anomaly detection model is a trained Bi-LSTM model based on the attention mechanism, and the training samples of the log anomaly detection model include the performance rules of historical logs and the historical dynamic event vectors of historical logs.

[0122] The embodiment of the present application adopts a Bi-LSTM model based on the attention mechanism for log anomaly detection, wherein Bi-LSTM uses two sets of LSTM networks with opposite directions to expand the single transmission direction, so that the dependencies of the log context can be obtained from two directions, and comprehensive context analysis can be achieved. The attention mechanism can also divide the importance of various dynamic events by weighting them with different weight values. The training process of the log anomaly detection model is as follows: Figure 3 As shown, it includes: processing historical logs to obtain historical static templates and historical dynamic parameters; vectorizing historical static templates and historical dynamic parameters respectively to obtain historical semantic vectors and historical dynamic parameter vectors; synthesizing historical semantic vectors and historical dynamic parameter vectors into historical dynamic event vectors; inputting the performance rules of historical logs and historical dynamic event vectors into the Bi-LSTM model based on the attention mechanism as training samples for training; when the Bi-LSTM model based on the attention mechanism meets the training conditions, a log anomaly detection model is obtained. The specific method for obtaining the historical semantic vector and the historical dynamic parameter vector is consistent with the principle of the method for obtaining the real-time semantic vector and the real-time dynamic parameter vector, and will not be repeated here. Since it is to realize log anomaly detection, the embodiment of the present application judges whether it is an abnormal log by the performance rules of the log, and the training samples of this log anomaly detection model are the performance rules of the historical log and the historical dynamic event vector of the historical log.

[0123] For example, the process of S104 Figure 3As shown, after obtaining the real-time dynamic event vector, the real-time dynamic event vector is input into the log anomaly detection model for context analysis to obtain the performance rules of the real-time log.

[0124] In this process, LogDection uses a Bi-LSTM neural network based on the attention mechanism. Bi-LSTM uses two sets of LSTM networks in opposite directions, which can mine the dependencies of log contexts from two directions, making the context analysis more comprehensive. LogDection can analyze the rules of dynamic event sequences to detect execution path anomalies and performance anomalies of smart terminal systems, thereby improving the monitoring capabilities of device logs, so as to grasp and predict the operation of smart terminal systems and reduce economic losses caused by equipment failures or downtime.

[0125] S105: If the performance pattern of the real-time log is consistent with the performance pattern of the abnormal log, the real-time log is detected to be abnormal.

[0126] After obtaining the performance rules of the real-time log, it is possible to determine which type of log the real-time log belongs to by extracting different performance rules of normal logs and abnormal logs from the historical log data. That is, if the performance rules of the real-time log are consistent with those of the abnormal log, the real-time log is detected to be abnormal; and if the performance rules of the real-time log are consistent with those of the normal log, the real-time log is detected to be a normal log.

[0127] Among them, after detecting anomalies in real-time logs, manual judgment can also be performed, specifically including: manually judging the real-time logs to obtain judgment results; if the judgment result is normal, it means that the performance rules obtained by using the log anomaly detection model have erroneous components, and the parameters of the log anomaly detection model need to be updated; if the judgment result is abnormal, the abnormal problem can be dealt with accordingly, thereby correcting the operation of the system.

[0128] In this process, different performance patterns of normal logs and abnormal logs can be extracted from historical log data, and then the log record sequence input in real time on the device side can be judged as abnormal based on the performance patterns, thereby improving the accuracy and efficiency of anomaly detection.

[0129] It can be seen that the embodiment of the present application can obtain the performance rules of the real-time log by using the log anomaly detection model to perform context analysis on the real-time log, and detect whether the current real-time log is an abnormal log based on the performance rules, thereby improving the accuracy and efficiency of anomaly detection. In addition, the real-time static template and real-time dynamic parameters of the input real-time log are decomposed, and the purpose of online log parsing can be achieved without waiting for the log batch collection, which lays the foundation for the system to detect anomalies in real time and respond quickly.

[0130] In order to make the method provided in the embodiment of the present application clearer and easier to understand, Figure 4 A specific example of this method is described.

[0131] like Figure 4 As shown, the implementation may include:

[0132] S401: Construct an n-gram template using historical logs.

[0133] The embodiment of the present application can adopt the n-gram template matching method to obtain static templates and dynamic parameters. Therefore, firstly, n-gram templates must be constructed based on historical logs. The specific method includes preprocessing historical logs to obtain historical log statements; extracting byte fragment sequences of length N that appear frequently from historical log statements; constructing a historical static template dictionary based on the byte fragment sequence of length N; and constructing n-gram templates based on the historical static template dictionary. That is, the LogParser tool can extract high-frequency 2-gram and 3-gram words from historical log data to construct a static template LogDict.

[0134] S402: Preprocess the real-time log to obtain real-time log statements.

[0135] In order to decompose the real-time static template and the real-time dynamic parameters, the real-time log needs to be preprocessed first to obtain the real-time log statements.

[0136] S403: Input the real-time log statement into the n-gram template for matching, and obtain the real-time static template and the real-time dynamic parameter.

[0137] This application realizes the decomposition of dynamic parameters of log static templates through n-gram template matching. Among them, LogParser uses a fixed-depth tree structure for matching. The child nodes of the fixed-depth parsing tree are only 2-gram and 3-gram nodes, and the parsing efficiency will not decrease due to the huge amount of data.

[0138] S404: vectorize the real-time static template to obtain a real-time semantic vector representing the specific log template.

[0139] Single static template matching cannot adapt to the changes in the upgrade of the intelligent terminal system. For this reason, it is necessary to convert the static template into an embedded vector and map the static template into a semantic vector of fixed dimension. The specific process may include: preprocessing the real-time static template and deleting the non-character part of the real-time static template; using the TF-IDF method to calculate the weight value of the word vector in the preprocessed real-time static template; finally, weighted addition of the word vector in the preprocessed real-time static template and the corresponding weight value to obtain a real-time semantic vector representing a specific log template. The specific calculation formula is shown in the above embodiment and will not be repeated here.

[0140] S405: vectorize the real-time dynamic parameters to obtain a real-time dynamic parameter vector.

[0141] Abnormal system performance will manifest as disordered changes in system parameters, and disordered parameters will be reflected in their vectorized form, so it is necessary to vectorize the log dynamic parameters. The specific process may include: using a fully connected neural network to obtain hidden information of real-time dynamic parameters; using a softmax function to calculate the hidden information of real-time dynamic parameters, the initialized weight vector and the bias vector to obtain a real-time dynamic parameter vector. The specific calculation formula is shown in the above embodiment and will not be repeated here.

[0142] S406: Synthesize the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector.

[0143] A single log can be represented as a semantic vector and a dynamic parameter vector. By connecting the real-time semantic vector with the real-time dynamic parameter vector, a dynamic event vector can be obtained. Then, log anomaly detection can be performed on the real-time log.

[0144] S407: Use the log anomaly detection model to perform context analysis on the real-time dynamic event vector to obtain the performance rules of the real-time log.

[0145] The embodiment of the present application adopts the Bi-LSTM model based on the attention mechanism for log anomaly detection, where Bi-LSTM uses two sets of LSTM networks with opposite directions to expand the single transmission direction, so that the dependency of the log context can be obtained from two directions to achieve comprehensive context analysis. The attention mechanism can also divide the importance of various dynamic events by weighting them with different weight values. Therefore, LogDection uses the Bi-LSTM model based on the attention mechanism as the anomaly detection framework, and its structure is as follows: Figure 5 shown.

[0146] At each time step, use the dynamic event vector V t and the passed hidden state h t Calculate the new cell state C t and the new hidden state ht-1 At time step t, the forward pass hidden vector and the backward pass hidden vector Concatenate into a new hidden vector h t , the specific formula is as follows:

[0147]

[0148] Among them, h t represents the new hidden vector; represents the forward pass hidden vector; represents the backward pass hidden vector.

[0149] The attention mechanism can automatically give low weights to insignificant noise by adding t The fully connected layer is added later. The weight α can measure the importance of the dynamic event to the overall sequence. The larger α is, the more important the dynamic event is. The calculation formula of weight α is as follows:

[0150]

[0151] Among them, α t represents the attention weight; W t α represents the weight at time step t; h t represents the new hidden vector.

[0152] Finally, the hidden state h t and the attention weight α t Multiply and accumulate, and use the softmax layer for classification. The specific formula is as follows:

[0153] pred=softmax(W'·h t )

[0154] Among them, pred represents the classification result; W' represents the softmax layer weight, and T represents the sequence length.

[0155] Using cross entropy as the loss function, the loss function calculation formula is as follows:

[0156]

[0157] in, represents the loss function; y (i) Represents the label of the i-th log in the sequence; Indicates the corresponding prediction results.

[0158] In this process, LogDection uses the Bi-LSTM model to mine the dependencies of log contexts from two directions, and can analyze the rules of dynamic event sequences to detect execution path anomalies and performance anomalies of the smart terminal system, thereby improving the monitoring capabilities of device logs, so as to grasp and predict the operation of the smart terminal system and reduce economic losses caused by equipment failures or downtime.

[0159] S408: If the performance pattern of the real-time log is consistent with the performance pattern of the normal log, the real-time log is detected as a normal log.

[0160] Different features of normal and abnormal logs can be extracted from historical log data, and then the log record sequence input in real time on the device side can be judged as abnormal based on these features. That is, if the performance pattern of the real-time log is consistent with that of the normal log, the real-time log is detected as a normal log.

[0161] S409: If the performance pattern of the real-time log is consistent with the performance pattern of the abnormal log, the real-time log is detected to be abnormal.

[0162] If the performance pattern of the real-time log is consistent with the performance pattern of the abnormal log, then the real-time log is detected to be abnormal. At this time, it is necessary to further verify the abnormal log detected by the model.

[0163] S410: Manually identify the real-time log to determine whether the identification result is abnormal.

[0164] That is, if an abnormal log is detected, the real-time log is manually judged to obtain the judgment result. In order to improve the accuracy of anomaly detection, after model detection, the detection result needs to be verified, that is, the real-time log needs to be manually judged, and the judgment result is used to determine whether the previous model detection result is correct. If the judgment result is abnormal, it means that the model detection result is correct, and S411 is executed; if the judgment result is normal, it means that the model detection result is wrong, and S412 needs to be executed to readjust the model.

[0165] S411: If the judgment result is abnormal, a corresponding treatment is adopted according to the abnormal problem.

[0166] S412: If the judgment result is normal, update the parameters of the log anomaly detection model.

[0167] This embodiment provides a log anomaly detection method based on deep learning, which uses a Bi-LSTM model to perform context analysis on real-time logs to obtain the performance rules of real-time logs, so that the log record sequence input in real time on the device side can be judged as abnormal based on the different features of normal and abnormal logs extracted from historical log data, thereby improving the accuracy and efficiency of anomaly detection.

[0168] See also Figure 6 , the embodiment of the present application provides a log anomaly detection device 600 based on deep learning, the device comprising:

[0169] An obtaining unit 601 is used to obtain a real-time static template and a real-time dynamic parameter according to the real-time log and the n-gram template, wherein the n-gram template is a statistical language model with a tree structure of a fixed depth;

[0170] A vectorization unit 602 is used to vectorize the real-time static template to obtain a real-time semantic vector representing a specific log template, and to vectorize the real-time dynamic parameter to obtain a real-time dynamic parameter vector;

[0171] A synthesis unit 603, used for synthesizing the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector;

[0172] An analysis unit 604 is used to perform context analysis on the real-time dynamic event vector using a log anomaly detection model to obtain a performance rule of the real-time log, wherein the log anomaly detection model is a trained Bi-LSTM model based on an attention mechanism, and the training samples of the log anomaly detection model include performance rules of historical logs and historical dynamic event vectors of historical logs;

[0173] The detection unit 605 is used to detect whether the real-time log is abnormal if the performance pattern of the real-time log is consistent with the performance pattern of the abnormal log.

[0174] Optionally, the obtaining unit 601 is specifically used for:

[0175] Preprocess the real-time log to obtain the real-time log statement;

[0176] The real-time log statements are input into the n-gram template for matching, and the real-time static templates matching the n-gram templates and the real-time dynamic parameters that cannot match the n-gram templates are obtained.

[0177] Optionally, the device 600 further includes a construction unit for:

[0178] Preprocess the historical logs to obtain historical log statements;

[0179] Extract the frequently occurring byte fragment sequence of length N from the historical log statements;

[0180] According to the byte segment sequence of length N, a historical static template dictionary is constructed;

[0181] Construct n-gram templates based on the historical static template dictionary.

[0182] Optionally, the vectorization unit 602 vectorizes the real-time static template to obtain a real-time semantic vector representing a specific log template, which is specifically used for:

[0183] Preprocess the real-time static template and delete the non-character part of the real-time static template;

[0184] Use the TF-IDF method to calculate the weight value of the word vector in the preprocessed real-time static template;

[0185] The word vectors in the preprocessed real-time static template are weighted and added to the corresponding weight values ​​to obtain a real-time semantic vector representing a specific log template.

[0186] Optionally, the vectorization unit 602 vectorizes the real-time dynamic parameter to obtain a real-time dynamic parameter vector, which is specifically used for:

[0187] Use fully connected neural networks to obtain hidden information of real-time dynamic parameters;

[0188] The softmax function is used to calculate the hidden information of the real-time dynamic parameters, the initialized weight vector and the bias vector to obtain the real-time dynamic parameter vector.

[0189] Optionally, the apparatus 600 further includes a training unit configured to:

[0190] Process historical logs to obtain historical static templates and historical dynamic parameters;

[0191] Vectorize the historical static template and the historical dynamic parameters respectively to obtain the historical semantic vector and the historical dynamic parameter vector;

[0192] The historical semantic vector and the historical dynamic parameter vector are synthesized into a historical dynamic event vector;

[0193] The performance rules of historical logs and historical dynamic event vectors are used as training samples and input into the Bi-LSTM model based on the attention mechanism for training;

[0194] When the Bi-LSTM model based on the attention mechanism meets the training conditions, a log anomaly detection model is obtained.

[0195] Optionally, after detecting that the real-time log is abnormal, the device 600 further includes:

[0196] A determination unit, used for manually determining the real-time log to obtain a determination result;

[0197] The processing unit is used to update the parameters of the log anomaly detection model if the judgment result is normal; if the judgment result is abnormal, adopt a response process according to the abnormal problem.

[0198] It should be noted that the specific implementation method and effect of the log anomaly detection device 600 based on deep learning can be found in the above Figure 1 or Figure 4 The relevant descriptions in the provided methods will not be repeated here.

[0199] The present application embodiment also provides an electronic device 700, such as Figure 7 As shown, the device 700 includes a memory 701 and a processor 702:

[0200] The memory 701 is used to store computer programs;

[0201] The processor 702 is used to execute the above Figure 1 or Figure 4 The method provided.

[0202] In addition, the present application also provides a computer-readable storage medium, the computer-readable storage medium is used to store a computer program, and the computer program is used to execute Figure 1 or Figure 4 The method provided.

[0203] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0204] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, in which the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the goals of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0205] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A log anomaly detection method based on deep learning, characterized in that: The method comprises: According to the real-time log and the n-gram template, a real-time static template and a real-time dynamic parameter are obtained, wherein the n-gram template is a statistical language model with a tree structure of a fixed depth; Vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template, and vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector; Combining the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector; Performing context analysis on the real-time dynamic event vector using a log anomaly detection model to obtain a performance rule of the real-time log, wherein the log anomaly detection model is a trained Bi-LSTM model based on an attention mechanism, and the training samples of the log anomaly detection model include performance rules of historical logs and historical dynamic event vectors of the historical logs; If the performance rule of the real-time log is consistent with the performance rule of the abnormal log, it is detected that the real-time log is abnormal.

2. The method according to claim 1, characterized in that The method of obtaining a real-time static template and a real-time dynamic parameter according to the real-time log and the n-gram template includes: Preprocessing the real-time log to obtain a real-time log statement; The real-time log statement is input into the n-gram template for matching, and a real-time static template matching the n-gram template and a real-time dynamic parameter that cannot match the n-gram template are obtained.

3. The method according to claim 1, characterized in that The process of constructing the n-gram template includes: Preprocess the historical logs to obtain historical log statements; Extracting a frequently occurring byte segment sequence of length N from the historical log statements; Constructing a historical static template dictionary according to the byte segment sequence of length N; The n-gram template is constructed according to the historical static template dictionary.

4. The method according to claim 1, characterized in that: The step of vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template includes: Preprocessing the real-time static template to delete non-character parts of the real-time static template; Use the TF-IDF method to calculate the weight value of the word vector in the preprocessed real-time static template; The word vectors in the preprocessed real-time static template and the corresponding weight values ​​are weightedly added to each other to obtain the real-time semantic vector representing the specific log template.

5. The method according to claim 1, characterized in that The vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector includes: Using a fully connected neural network to obtain hidden information of the real-time dynamic parameters; The hidden information of the real-time dynamic parameter, the initialized weight vector and the bias vector are calculated using a softmax function to obtain the real-time dynamic parameter vector.

6. The method according to claim 1, characterized in that The training process of the log anomaly detection model includes: Process historical logs to obtain historical static templates and historical dynamic parameters; Vectorizing the historical static template and the historical dynamic parameter respectively to obtain the historical semantic vector and the historical dynamic parameter vector; Combining the historical semantic vector and the historical dynamic parameter vector into a historical dynamic event vector; Inputting the performance rules of the historical logs and the historical dynamic event vectors as training samples into a Bi-LSTM model based on an attention mechanism for training; When the attention mechanism-based Bi-LSTM model meets the training conditions, the log anomaly detection model is obtained.

7. The method according to claim 1, characterized in that After detecting that the real-time log is abnormal, the method further includes: Manually identify the real-time log to obtain an identification result; If the judgment result is normal, updating the parameters of the log anomaly detection model; If the determination result is abnormal, a response is taken according to the abnormal problem.

8. A log anomaly detection device based on deep learning, characterized in that: The device comprises: An obtaining unit, used for obtaining a real-time static template and a real-time dynamic parameter according to a real-time log and an n-gram template, wherein the n-gram template is a statistical language model with a tree structure of a fixed depth; A vectorization unit, used for vectorizing the real-time static template to obtain a real-time semantic vector representing a specific log template, and vectorizing the real-time dynamic parameter to obtain a real-time dynamic parameter vector; A synthesis unit, used for synthesizing the real-time semantic vector and the real-time dynamic parameter vector into a real-time dynamic event vector; An analysis unit, configured to perform context analysis on the real-time dynamic event vector using a log anomaly detection model to obtain a performance rule of the real-time log, wherein the log anomaly detection model is a trained Bi-LSTM model based on an attention mechanism, and the training samples of the log anomaly detection model include performance rules of historical logs and historical dynamic event vectors of the historical logs; The detection unit is used to detect that an abnormality occurs in the real-time log if the performance pattern of the real-time log is consistent with the performance pattern of the abnormal log.

9. An electronic device, characterized in that: The device comprises a memory and a processor, and the electronic device is used to execute a program stored in the memory to run the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

Images