Log processing method, apparatus, device, and medium
By integrating learning strategies and LSTM and VAE models to perform anomaly detection and root cause localization on slice logs, the accuracy problem of log root cause localization in field networks is solved, and efficient multi-anomaly detection and root cause identification are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2023-03-23
- Publication Date
- 2026-08-04
AI Technical Summary
Existing log-based root cause localization methods have poor accuracy and generalization ability in industry field networks, and are difficult to detect anomalies not in the expert knowledge base.
An ensemble learning strategy is employed to detect anomalies in sliced logs. By combining LSTM and VAE models to learn the context sequence and implicit distribution information of the logs, root cause localization is performed through semantic information.
It improves the accuracy and recall of log anomaly detection, can identify multiple root causes, has strong generalization ability, and a high degree of intelligence. It is more accurate and efficient than expert operation and maintenance knowledge methods.
Smart Images
Figure CN116910009B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication technology, and in particular to a log processing method, apparatus, device, and medium. Background Technology
[0002] Industry field network devices generate a large number of logs during operation. These logs record the system's running status and various operations performed, serving as a valuable source of information for online monitoring and anomaly detection. Therefore, quickly and accurately detecting abnormal logs within the system is crucial for maintaining system security and stability. However, using logs for field network operation and maintenance presents challenges. Industry field networks are characterized by complex networking, a large number of devices, and a high degree of virtualization. The massive amounts of logs generated during device operation, coupled with inconsistent standards and unstructured nature, make manual analysis relying on expert experience inefficient and inaccurate.
[0003] Currently, root cause analysis methods based on logs mainly rely on expert operational knowledge (keywords or rules) to locate root causes. However, this method is limited by the coverage of expert knowledge, and it cannot detect anomalies that are not in the expert knowledge base, resulting in poor accuracy and generalization ability. Summary of the Invention
[0004] The purpose of this invention is to provide a log processing method, apparatus, device, and medium that solves the problem of low accuracy in detection results and root cause localization in existing network anomaly detection methods.
[0005] To achieve the above objectives, embodiments of the present invention provide a log processing method, the method comprising:
[0006] An ensemble learning strategy is used to detect anomalies in the slice logs, and the detection results of the slice logs are obtained. The ensemble learning strategy is used to detect various anomalies in the slice logs.
[0007] If the detection result indicates that the slice log is abnormal, the root cause of the slice log is located based on the semantic information of the slice log.
[0008] Optionally, the anomaly detection includes at least one of the following:
[0009] Sequence anomaly detection;
[0010] Single-point anomaly detection;
[0011] Quantitative anomaly detection.
[0012] Optionally, the step of using an ensemble learning strategy to perform anomaly detection on the sliced logs and obtaining the detection results of the sliced logs includes:
[0013] Based on the context sequence information of the segmented log, sequence anomaly detection is performed on the segmented log to obtain a first detection result vector of the segmented log;
[0014] The implicit distribution information of the segment log is obtained based on the feature data of the segment log, and anomaly detection is performed on the segment log based on the implicit distribution information to obtain the second detection result vector of the segment log;
[0015] The first detection result vector and the second detection result vector are concatenated to obtain the third detection result vector.
[0016] Anomaly detection is performed on the slice log based on the third detection result vector to obtain the detection result of the slice log.
[0017] Optionally, the step of performing sequence anomaly detection on the segmented log based on the context sequence information of the segmented log to obtain a first detection result vector of the segmented log includes:
[0018] The context sequence information of the slice log is learned by a Long Short-Term Memory (LSTM) network model, and the slice log is subjected to sequence anomaly detection based on the context sequence information to obtain the first detection result vector of the slice log.
[0019] The step of performing anomaly detection on the slice log based on the implicit distribution information to obtain a second detection result vector of the slice log includes:
[0020] The hidden distribution information of the segmented log is learned by a variational autoencoder (VAE), and anomaly detection is performed on the segmented log based on the feature data of the segmented log to obtain a second detection result vector of the segmented log.
[0021] The step of performing anomaly detection on the segment log based on the third detection result vector to obtain the detection result of the segment log includes:
[0022] The vector information in the third detection result vector is fused using a VAE model to learn the hidden distribution information of the third detection result vector. Anomaly detection is then performed on the slice log based on the hidden distribution information to obtain the detection result of the slice log.
[0023] Optionally, before using an ensemble learning strategy to perform anomaly detection on the sliced logs, the method further includes:
[0024] LSTM model training is performed based on training set data to learn the implicit context sequence information in the training set data and obtain the LSTM prediction probability vector.
[0025] VAE model training is performed based on training set data to learn the implicit distribution of training set data and obtain VAE prediction feature anomaly vectors.
[0026] The LSTM prediction probability vector and the VAE prediction feature anomaly vector are concatenated to obtain a second vector.
[0027] The second vector is input into the VAE model for model training to obtain an ensemble learning model.
[0028] Optionally, before using an ensemble learning strategy to perform anomaly detection on the sliced logs, the method further includes:
[0029] Perform log preprocessing on the logs to be detected;
[0030] The preprocessed logs to be detected are parsed to obtain a structured log template with event templates and specific parameters;
[0031] Feature extraction is performed on the structured log template to obtain at least one slice log corresponding to the log to be detected.
[0032] Optionally, the step of performing root cause localization on the slice logs based on the semantic information of the slice logs includes:
[0033] Each log text information corresponding to the sliced log is input into the text vector model, and the semantic information corresponding to each log text information is extracted. The semantic information forms a text vector.
[0034] The text vector is input into the semantic model to perform root cause localization on the slice log.
[0035] Optionally, before performing root cause localization on the slice logs based on the semantic information of the slice logs, the method further includes:
[0036] The text vector model is trained using the training set data, and the semantic information of the log text information in the training set data is extracted to form the text vector of the training set data.
[0037] A semantic model is trained using the text vectors from the training set data;
[0038] The trained semantic model is used to determine whether the log expresses positive or negative information, and to identify the root cause of the log behavior when the log expresses negative information.
[0039] To achieve the above objectives, embodiments of the present invention provide a log processing apparatus, the apparatus comprising:
[0040] The log anomaly detection module is used to perform anomaly detection on the slice log using an ensemble learning strategy to obtain the detection result of the slice log. The ensemble learning strategy is used to detect various anomalies in the slice log.
[0041] The log root cause localization module is used to perform root cause localization on the segment log based on the semantic information of the segment log when the detection result indicates that the segment log is abnormal.
[0042] To achieve the above objectives, embodiments of the present invention provide a log processing device, including a processor and a transceiver;
[0043] The processor is configured to: perform anomaly detection on the slice logs using an ensemble learning strategy, and obtain the detection results of the slice logs, wherein the ensemble learning strategy is used to detect various anomalies in the slice logs;
[0044] If the detection result indicates that the slice log is abnormal, the root cause of the slice log is located based on the semantic information of the slice log.
[0045] To achieve the above objectives, embodiments of the present invention provide a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps of the above-described log processing method.
[0046] The beneficial effects of the above-described technical solution of the present invention are as follows:
[0047] The embodiments of this application employ an ensemble learning strategy to perform anomaly detection on segmented logs, enabling the simultaneous detection of multiple anomalies and improving the accuracy and recall of log anomaly detection. When the detection result indicates an anomaly in the segmented log, root cause localization is performed based on the semantic information of the segmented log. This fully utilizes the semantic information expressed in the log, enabling the identification of multiple root causes with higher accuracy, stronger generalization ability, and a higher degree of intelligence. Attached Figure Description
[0048] Figure 1 This is one of the flowcharts illustrating the log processing method according to an embodiment of the present invention;
[0049] Figure 2 This is a schematic diagram of log parsing according to an embodiment of the present invention;
[0050] Figure 3 This is a schematic diagram of log feature extraction according to an embodiment of the present invention;
[0051] Figure 4This is a schematic diagram of the process for anomaly detection in slice logs in an embodiment of the present invention;
[0052] Figure 5 This is a schematic diagram of the model training for log anomaly detection in an embodiment of the present invention;
[0053] Figure 6 This is a schematic diagram of the model training for root cause localization in an embodiment of the present invention;
[0054] Figure 7 This is a schematic diagram of the output of the semantic VAE model in an embodiment of the present invention;
[0055] Figure 8 This is a schematic diagram of the root cause localization process according to an embodiment of the present invention;
[0056] Figure 9 This is a second schematic flowchart of the log processing method according to an embodiment of the present invention;
[0057] Figure 10 This is a schematic diagram of the log processing device according to an embodiment of the present invention;
[0058] Figure 11 This is one of the structural schematic diagrams of the log processing device according to an embodiment of the present invention;
[0059] Figure 12 This is a second schematic diagram of the log processing device according to an embodiment of the present invention. Detailed Implementation
[0060] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0061] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.
[0062] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0063] In addition, the terms "system" and "network" are often used interchangeably in this article.
[0064] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.
[0065] like Figure 1 As shown, an embodiment of the present invention provides a log processing method, including:
[0066] Step 101: Use an ensemble learning strategy to perform anomaly detection on the slice logs and obtain the detection results of the slice logs. The ensemble learning strategy is used to detect various anomalies in the slice logs.
[0067] Step 102: If the detection result indicates that the slice log is abnormal, perform root cause localization on the slice log based on the semantic information of the slice log.
[0068] In this embodiment, log processing can include two parts: log anomaly detection and root cause localization. First, anomaly detection is performed using slices of the log data to be detected. If anomalies are detected in a slice, root cause localization is then performed. Specifically, when detecting anomalies in sliced logs, an ensemble learning strategy can be used to simultaneously detect multiple anomalies in the sliced logs. Optionally, the ensemble learning strategy can integrate the advantages of the LSTM algorithm in context sequence learning and the advantages of VAE in single-point anomaly and quantitative anomaly detection, thereby achieving the detection of multiple log anomalies such as sequence anomalies and single-point anomalies, improving the accuracy and recall of log anomaly detection.
[0069] Optionally, the anomaly detection includes at least one of the following:
[0070] Sequence anomaly detection;
[0071] Single-point anomaly detection;
[0072] Quantitative anomaly detection.
[0073] In this embodiment, an ensemble learning strategy can be used to simultaneously detect multiple anomalies in the sliced logs, such as multiple anomalies including sequence anomalies, single-point anomalies, and quantitative anomalies. That is, the obtained detection results integrate multiple detection results such as sequence anomalies, single-point anomalies, and quantitative anomalies. It should be noted that the anomaly detection may also include other anomaly detections in the sliced logs; this is only an illustrative example and not a specific limitation.
[0074] The data used in this embodiment is log data. The training set used for training models related to anomaly detection or root cause localization is the logs during normal system operation, and the test set for anomaly detection or root cause localization is the logs generated by the equipment in actual production, including normal logs and some abnormal logs. These logs record the system's running status, performance indicators, and various operations performed by the system.
[0075] In industry field networks, log-based anomalies mainly include sequence anomalies, single-point anomalies, and quantitative anomalies. Sequence anomalies, also known as operational mode anomalies, refer to a program execution process with a certain logical order, and the corresponding log entries also have a sequential order, such as 1-2-3-4. If the actual log entry is 1-2-4-3, then it is a sequence anomaly. Single-point anomalies are anomalies that can be reflected by a single log entry, such as the "Error 400" log entry. Quantitative anomalies refer to a normal log sequence that should have a definite relationship. For example, if there are log entries for opening files, there must also be corresponding log entries for closing files, and the number of both should be roughly equal; otherwise, it is considered a quantitative anomaly.
[0076] The log-based anomaly detection process involves identifying an anomaly in the current log slice sequence, followed by root cause localization. This involves determining which specific log line caused the anomaly, clarifying the cause, and then fixing the anomaly based on that cause. This embodiment incorporates semantic information into root cause localization. Semantic learning models capable of sentence semantic reasoning and learning log statement semantic information, such as Doc2Vec (Document to Vector), tex2vec, and Word2vec, can be selected, without limitation. Therefore, this solution fully utilizes the semantic information expressed in log statements for root cause localization, achieving higher accuracy, stronger generalization ability, and a higher degree of intelligence compared to existing root cause localization methods based on expert operational knowledge.
[0077] The embodiments of this application employ an ensemble learning strategy to perform anomaly detection on segmented logs, enabling the simultaneous detection of multiple anomalies and improving the accuracy and recall of log anomaly detection. When the detection result indicates an anomaly in the segmented log, root cause localization is performed based on the semantic information of the segmented log. This fully utilizes the semantic information expressed in the log, enabling the identification of multiple root causes with higher accuracy, stronger generalization ability, and a higher degree of intelligence.
[0078] As an optional embodiment, before performing anomaly detection on the sliced logs using an ensemble learning strategy, the method further includes:
[0079] Step 21: Perform log preprocessing on the logs to be detected.
[0080] This step mainly involves preprocessing to address issues such as inconsistent log formats. This preprocessing includes, for example, automatically identifying HTTP addresses, dates, image names, file names, IP addresses, and other information that needs to be preprocessed before template extraction, based on the characteristics of the log messages. It also involves retaining important information from the logs, removing irrelevant information, and maintaining the consistency of log formats.
[0081] Step 22: Parse the preprocessed logs to be detected to obtain a structured log template with event templates and specific parameters.
[0082] Based on the preprocessed logs, log parsing algorithms (such as Drain and Spell algorithms) are used to automatically extract event templates. That is, the input is the raw, unstructured log message, and the output is a structured form with event templates (constants) and specific parameters (variables). For example, log parsing... Figure 2 As shown, each log template corresponds to a unique ID.
[0083] Step 23: Extract features from the structured log template to obtain at least one slice log corresponding to the log to be detected.
[0084] The structured log templates obtained after log parsing are further encoded into numerical feature vectors. In this embodiment, the entire training log after parsing is divided into shorter log slices, the length n of which depends on the specific scenario. Each log entry is represented by its corresponding log template identifier (Identity Document, ID). Log feature extraction, for example... Figure 3 As shown. The log slice sequence is then used as input for subsequent anomaly detection algorithms.
[0085] As an optional embodiment, the step of using an ensemble learning strategy to perform anomaly detection on the segmented logs and obtaining the detection results of the segmented logs includes:
[0086] Based on the context sequence information of the segmented log, sequence anomaly detection is performed on the segmented log to obtain a first detection result vector of the segmented log;
[0087] The implicit distribution information of the segment log is obtained based on the feature data of the segment log, and anomaly detection is performed on the segment log based on the implicit distribution information to obtain the second detection result vector of the segment log;
[0088] The first detection result vector and the second detection result vector are concatenated to obtain the third detection result vector.
[0089] Anomaly detection is performed on the segmented log based on the third detection result vector to obtain the detection result of the segmented log. Optionally, the step of performing sequence anomaly detection on the segmented log based on the context sequence information of the segmented log to obtain the first detection result vector of the segmented log includes:
[0090] The context sequence information of the slice log is learned by the Long Short-Term Memory (LSTM) network model, and the slice log is subjected to sequence anomaly detection based on the context sequence information to obtain the first detection result vector of the slice log.
[0091] The step of performing anomaly detection on the slice log based on the implicit distribution information to obtain a second detection result vector of the slice log includes:
[0092] The hidden distribution information of the segment log is learned by the variational autoencoder (VAE), and anomaly detection is performed on the segment log based on the feature data of the segment log to obtain the second detection result vector of the segment log.
[0093] The step of performing anomaly detection on the segment log based on the third detection result vector to obtain the detection result of the segment log includes:
[0094] The vector information in the third detection result vector is fused using a VAE model to learn the hidden distribution information of the third detection result vector. Anomaly detection is then performed on the slice log based on the hidden distribution information to obtain the detection result of the slice log.
[0095] In this embodiment, sequence anomaly detection can be performed on the slice logs based on a trained LSTM model. It should be noted that other models capable of sequence anomaly detection can also be used to perform sequence anomaly detection on the slice logs, and this is not limited here.
[0096] Optionally, the second detection result vector can be a single-point anomaly, quantitative anomaly, or other detection. This embodiment obtains the implicit distribution information of the log data based on a trained VAE model, thereby detecting single-point and quantitative anomalies in the sliced log. It should be noted that other models capable of single-point and quantitative anomaly detection can also be used to perform single-point and quantitative anomaly detection on the sliced log, and this is not limited here.
[0097] Optionally, ensemble detection results for sliced logs can be obtained based on a pre-trained ensemble learning model. The process of anomaly detection for sliced logs is as follows: Figure 4 As shown, the ensemble learning model can be an ensemble learning VAE model.
[0098] Based on a pre-trained LSTM model, VAE model, and ensemble learning model, the segmented log sequence to be detected is input into the LSTM model to obtain an LSTM probability vector (i.e., the first detection result vector), and the segmented log sequence to be detected is input into the VAE model to obtain a VAE feature outlier vector (i.e., the second detection result vector). These two vectors are concatenated (i.e., the third detection result vector) and input into the ensemble learning model for anomaly detection, outputting the anomaly detection results for the segmented log. This fully utilizes the advantages of the LSTM algorithm in context sequence learning and the advantages of VAE in single-point anomaly and quantitative anomaly detection.
[0099] In this embodiment, the hidden information of the LSTM hidden layer and the hidden information and features extracted by the fully connected layer of the VAE module are combined by integrating the VAE learning model. By comprehensively utilizing the advantages of the bidirectional LSTM algorithm in context sequence learning and the advantages of VAE in single-point anomaly and quantitative anomaly detection, the detection of various log anomalies such as sequence anomalies and single-point anomalies is realized, thereby improving the accuracy and recall of log anomaly detection.
[0100] As an optional embodiment, before performing anomaly detection on the sliced logs using an ensemble learning strategy, the method further includes:
[0101] Step 31: Train the LSTM model based on the training set data, learn the context sequence information hidden in the training set data, and obtain the LSTM prediction probability vector.
[0102] In this embodiment, the process of training the model for log anomaly detection based on the ensemble learning strategy is as follows: Figure 5 As shown. A bidirectional LSTM model can be trained based on feature data (sliced sequences) to learn the contextual sequence information implicit in the logs. The output here can be hidden layer information or output prediction probability information; this embodiment uses output probability information as an example.
[0103] The output LSTM prediction probability vector is: A j = [1-a1, 1-a2, ..., 1-a] n-w ].
[0104] Where j is the slice sequence value, n is the slice sequence length, w is the size of the bidirectional LSTM sliding window, and a i Let a be the probability that the LSTM predicts the current real log template ID when it slides to the i-th window. iThe smaller the value, the more abnormal it is. For example, when the actual log sequence is [1,2,3,4,5,6,7,8,9,10], and the sliding window size is 3, then a1 is the probability that the bidirectional LSTM predicts the next sequence as 4 based on the first sliding window sequence [1,2,3]. 1-a1 is the degree of abnormality of the current sequence being predicted as [1,2,3,4]. The larger the value, the more abnormal it is.
[0105] Step 32: Train the VAE model based on the training set data, learn the implicit distribution information of the training set data, and obtain the VAE prediction feature anomaly vector.
[0106] A VAE model may include: an encoder, a mean calculation unit, a standard deviation calculation unit, a unit for determining the latent distribution, and a decoder.
[0107] The VAE model is trained based on feature data (slice sequences) to learn the latent distribution of the slice sequences. Here, the decoded information of the fully connected output is used, namely the VAE feature anomaly vector B. j =[b1,b2,……,b n ], j is the slice sequence value, n is the slice sequence length, b i This is used to predict the probability that the i-th template is anomalous in the VAE. The larger the value, the more anomalous it is.
[0108] Step 33: Concatenate the LSTM prediction probability vector and the VAE prediction feature anomaly vector to obtain the concatenated vector;
[0109] Step 34: Input the concatenated vector into the VAE model for model training to obtain the ensemble learning model. The ensemble learning model can be an ensemble learning VAE model.
[0110] In this embodiment, the LSTM prediction probability vector A j = [1-a1, 1-a2, ..., 1-a] n-w ] and VAE predicted feature anomaly vector B j =[b1,b2,……,b n Concatenate the vectors to form vector C. j =[a1,a2,……,a n-w b1, b2, ..., b n The vector C is the concatenated vector, and vector C is input into the VAE model for model training.
[0111] By integrating the learning strategies of the three models mentioned above, we can utilize deep hidden features and information to comprehensively leverage the advantages of the bidirectional LSTM algorithm in context sequence learning and the advantages of VAE in detecting single-point anomalies, quantitative anomalies, etc. This approach can detect a variety of anomalies and improve the accuracy and recall of log anomaly detection.
[0112] As an optional embodiment, the root cause localization of the segment logs based on the semantic information of the segment logs includes:
[0113] Each log text information corresponding to the segmented log is input into a text vector model to extract semantic information corresponding to each log text information, and the semantic information forms a text vector; the text vector is input into a semantic model to perform root cause localization on the segmented log.
[0114] Optionally, before performing root cause localization on the slice logs based on the semantic information of the slice logs, the method further includes:
[0115] The text vector model is trained using the training set data, and the semantic information of the log text information in the training set data is extracted to form the text vector of the training set data.
[0116] A semantic model is trained using the text vectors from the training set data;
[0117] The trained semantic model is used to determine whether the log expresses positive or negative information, and to identify the root cause of the log behavior when the log expresses negative information.
[0118] like Figure 6 As shown, in this embodiment, taking the text vector model as a Doc2vec model and the semantic model as a semantic VAE model as an example, the Doc2vec model is trained based on the preprocessed log text information of the training set. Here, words that have not appeared are uniformly replaced with specific strings to facilitate subsequent text vector reasoning and expansion. Semantic information is extracted from the logs to form Doc2Vec vectors. Using the log Doc2vec vectors of the training set as input, the semantic VAE model is trained to extract the distribution information of semantic expressions. The trained semantic VAE model is used to determine the positive or negative information of the log expression semantics. If it is a negative semantic expression, then the root cause of the log behavior is determined, such as... Figure 7 As shown.
[0119] Taking the Doc2vec model as the text vector model and the semantic VAE model as the semantic model as an example, when root cause localization is performed after determining log anomalies, the semantic-based root cause localization process is as follows: Figure 8As shown, if a segmented log sequence is detected as an anomaly, the text information of each corresponding log entry in the segmented log is input into the trained Doc2vec model to infer the Doc2vec vector corresponding to each log entry. Then, root cause localization is performed based on the trained semantic VAE model, and the root cause localization result is output. Segmented logs are detected as anomalies, such as one or more of the following: sequence anomalies, single-point anomalies, quantitative anomalies, etc., and may also include other anomalies.
[0120] In this embodiment, semantic information is incorporated into root cause localization. Based on the characteristics of the logs, a semantic learning model capable of performing sentence semantic reasoning and learning the semantic information of log statements, such as Doc2Vec, tex2vec, and Word2vec, is selected for training the semantic vector model. This embodiment is not limited to using only Doc2vec. Then, a semantic VAE model is trained based on the semantic information stored in the semantic vectors, ultimately achieving root cause localization of the logs. Therefore, this embodiment fully utilizes the semantic information expressed in the log statements for root cause localization, resulting in higher accuracy, stronger generalization ability, and a higher degree of intelligence compared to root cause localization methods based on expert operational knowledge.
[0121] The log processing method in this application includes two parts: log anomaly detection and log root cause localization, such as... Figure 9 As shown, taking sequence anomaly detection using an LSTN model, single-point anomaly detection using a VAE model, and quantitative anomaly detection as examples, the log anomaly detection process mainly includes four steps: log preprocessing, log parsing, feature extraction, and log anomaly detection based on an ensemble learning strategy. Specifically, log anomaly detection based on an ensemble learning strategy can include calculating bidirectional LSTM probability vectors and VAE feature outlier vectors. The bidirectional LSTM probability vector and the VAE feature outlier vector are then concatenated and input into the ensemble learning VAE model to obtain the anomaly detection result.
[0122] The log root cause localization process mainly includes: first, training a text vector model (e.g., Doc2vec model) based on the preprocessed logs to extract text vectors (e.g., Doc2vec vectors). Then, training a semantic VAE model based on the semantic information stored in the text vectors, and finally achieving root cause localization of the logs based on the trained semantic model.
[0123] The embodiments of this application can be applied to a lightweight operation and maintenance platform for 5G private networks, which can realize anomaly detection and root cause localization of 5G private networks based on the logs of network devices such as 5G gateways and modules collected by the platform.
[0124] The embodiments of this application can simultaneously achieve anomaly detection and root cause localization, improving the accuracy and recall of anomaly detection, the accuracy and generalization ability of root cause localization, and exhibiting a high degree of intelligence. The anomaly detection method based on an ensemble learning strategy (simultaneously detecting multiple anomalies in sliced logs) can detect various log anomalies such as sequence anomalies, single-point anomalies, and quantitative anomalies, improving the accuracy, recall, and generalization ability of log anomaly detection. In the semantic-based root cause localization method, a semantic model is trained based on the semantic information stored in the text vector, ultimately achieving root cause localization of the logs. This method fully utilizes the semantic information expressed in the logs, enabling the identification of multiple root causes, and achieving higher accuracy, stronger generalization ability, and a higher degree of intelligence compared to root cause localization methods based on expert operational knowledge. This application can accurately predict field network anomalies and find the root causes of faults based on anomaly detection results, improving operational efficiency.
[0125] like Figure 10 As shown, this embodiment of the invention also provides a log processing device 1000, comprising:
[0126] The log anomaly detection module 1010 is used to perform anomaly detection on the slice log using an ensemble learning strategy to obtain the detection result of the slice log. The ensemble learning strategy is used to detect various anomalies in the slice log.
[0127] The log root cause localization module 1020 is used to perform root cause localization on the segment log based on the semantic information of the segment log when the detection result indicates that the segment log is abnormal.
[0128] Optionally, the anomaly detection includes at least one of the following:
[0129] Sequence anomaly detection;
[0130] Single-point anomaly detection;
[0131] Quantitative anomaly detection.
[0132] Optionally, the log anomaly detection module 1010 includes:
[0133] The first detection unit 1011 is used to perform sequence anomaly detection on the slice log based on the context sequence information of the slice log, and obtain a first detection result vector of the slice log;
[0134] The second detection unit 1012 is used to obtain the implicit distribution information of the slice log based on the feature data of the slice log, perform anomaly detection on the slice log based on the implicit distribution information, and obtain a second detection result vector of the slice log.
[0135] The integrated learning unit 1013 is used to concatenate the first detection result vector and the second detection result vector to obtain a third detection result vector; and to perform anomaly detection on the slice log based on the third detection result vector to obtain the detection result of the slice log.
[0136] Optionally, the first detection unit is specifically used to: learn the context sequence information of the slice log through a Long Short-Term Memory (LSTM) network model, and perform sequence anomaly detection on the slice log based on the context sequence information to obtain a first detection result vector of the slice log;
[0137] The second detection unit is specifically used to: learn the implicit distribution information of the slice log through a variational autoencoder (VAE), and perform anomaly detection on the slice log based on the feature data of the slice log to obtain a second detection result vector of the slice log;
[0138] The integrated learning unit is specifically used to: fuse the vector information in the third detection result vector through the VAE model, learn the hidden distribution information of the third detection result vector, perform anomaly detection on the slice log based on the hidden distribution information, and obtain the detection result of the slice log.
[0139] Optionally, the device further includes:
[0140] The preprocessing module is used to preprocess the logs to be detected.
[0141] The parsing module is used to parse the preprocessed logs to be detected and obtain a structured log template with event templates and specific parameters.
[0142] The feature extraction module is used to extract features from the structured log template to obtain at least one slice log corresponding to the log to be detected.
[0143] Optionally, the log root cause localization module 1020 includes:
[0144] The semantic vector learning unit 1021 is used to input each log text information corresponding to the sliced log into the text vector model, extract the semantic information corresponding to each log text information, and form a text vector from the semantic information.
[0145] The semantic model root cause localization unit 1022 is used to input the text vector into the semantic model to perform root cause localization on the slice log.
[0146] Optionally, the device further includes:
[0147] The fourth training module is used to train a text vector model using the training set data, extract the semantic information of the log text information in the training set data, and form the text vector of the training set data.
[0148] The fifth training module is used to train a semantic model using the text vectors of the training set data;
[0149] The trained semantic model is used to determine whether the log expresses positive or negative information, and to identify the root cause of the log behavior when the log expresses negative information.
[0150] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.
[0151] like Figure 11 As shown, a log processing device 1100 according to an embodiment of the present invention includes a processor 1110 and a transceiver 1120, wherein,
[0152] The processor 1110 is configured to: perform anomaly detection on the slice log using an ensemble learning strategy, and obtain the detection result of the slice log, wherein the ensemble learning strategy is used to detect various anomalies in the slice log;
[0153] If the detection result indicates that the slice log is abnormal, the root cause of the slice log is located based on the semantic information of the slice log.
[0154] Optionally, the anomaly detection includes at least one of the following:
[0155] Sequence anomaly detection;
[0156] Single-point anomaly detection;
[0157] Quantitative anomaly detection.
[0158] Optionally, the processor uses an ensemble learning strategy to perform anomaly detection on the slice logs and obtains the detection results of the slice logs, including:
[0159] Based on the context sequence information of the segmented log, sequence anomaly detection is performed on the segmented log to obtain a first detection result vector of the segmented log;
[0160] The implicit distribution information of the segment log is obtained based on the feature data of the segment log, and anomaly detection is performed on the segment log based on the implicit distribution information to obtain the second detection result vector of the segment log;
[0161] The first detection result vector and the second detection result vector are concatenated to obtain the third detection result vector.
[0162] Anomaly detection is performed on the slice log based on the third detection result vector to obtain the detection result of the slice log.
[0163] Optionally, the processor performs sequence anomaly detection on the slice log based on the context sequence information of the slice log to obtain a first detection result vector of the slice log, including:
[0164] The context sequence information of the slice log is learned by the Long Short-Term Memory (LSTM) network model, and the slice log is subjected to sequence anomaly detection based on the context sequence information to obtain the first detection result vector of the slice log.
[0165] The processor performs anomaly detection on the slice log based on the implicit distribution information to obtain a second detection result vector of the slice log, including:
[0166] The hidden distribution information of the segment log is learned by the variational autoencoder (VAE), and anomaly detection is performed on the segment log based on the feature data of the segment log to obtain the second detection result vector of the segment log.
[0167] The processor performs anomaly detection on the slice log based on the third detection result vector, and obtains the detection result of the slice log including:
[0168] The vector information in the third detection result vector is fused using a VAE model to learn the hidden distribution information of the third detection result vector. Anomaly detection is then performed on the slice log based on the hidden distribution information to obtain the detection result of the slice log.
[0169] Optionally, the processor is further configured to:
[0170] Perform log preprocessing on the logs to be detected;
[0171] The preprocessed logs to be detected are parsed to obtain a structured log template with event templates and specific parameters;
[0172] Feature extraction is performed on the structured log template to obtain at least one slice log corresponding to the log to be detected.
[0173] Optionally, the processor performs root cause localization on the slice logs based on the semantic information of the slice logs, including:
[0174] Each log text information corresponding to the sliced log is input into the text vector model, and the semantic information corresponding to each log text information is extracted. The semantic information forms a text vector.
[0175] The text vector is input into the semantic model to perform root cause localization on the slice log.
[0176] Optionally, the processor is further configured to:
[0177] The text vector model is trained using the training set data, and the semantic information of the log text information in the training set data is extracted to form the text vector of the training set data.
[0178] A semantic model is trained using the text vectors from the training set data;
[0179] The trained semantic model is used to determine whether the log expresses positive or negative information, and to identify the root cause of the log behavior when the log expresses negative information.
[0180] It should be noted that the log processing device provided in the embodiments of the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.
[0181] Another embodiment of the log processing device of the present invention, such as Figure 12 As shown, it includes a transceiver 1210, a processor 1200, a memory 1220, and a program or instructions stored in the memory 1220 and executable on the processor 1200; when the processor 1200 executes the program or instructions, it implements the above-mentioned log processing method.
[0182] The transceiver 1210 is used to receive and send data under the control of the processor 1200.
[0183] Among them, Figure 12 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 1200) and memory (memory 1220). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1210 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 1200 is responsible for managing the bus architecture and general processing, and the memory 1220 may store data used by the processor 1200 during operation.
[0184] An embodiment of the present invention provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the log processing method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.
[0185] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0186] It should be further noted that the electronic devices described in this specification include, but are not limited to, smartphones, tablets, etc., and many of the described functional components are referred to as modules in order to more specifically emphasize the independence of their implementation.
[0187] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.
[0188] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist at least in part as electronic signals within the system or network.
[0189] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.
[0190] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values includes the upper and lower limits of the range and any subranges in between.
[0191] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A log processing method characterized by, include: An ensemble learning strategy is used to detect anomalies in the slice logs, and the detection results of the slice logs are obtained. The ensemble learning strategy is used to detect various anomalies in the slice logs. If the detection result indicates that the slice log is abnormal, the root cause of the slice log is located based on the semantic information of the slice log. The step of using an ensemble learning strategy to perform anomaly detection on the segmented logs and obtaining the detection results of the segmented logs includes: Based on the context sequence information of the segmented log, sequence anomaly detection is performed on the segmented log to obtain a first detection result vector of the segmented log; The implicit distribution information of the segmented logs is obtained based on the feature data of the segmented logs. Anomaly detection is performed on the segmented logs based on the implicit distribution information to obtain a second detection result vector of the segmented logs. The second detection result is single-point anomaly detection and / or quantitative anomaly detection. The first detection result vector and the second detection result vector are concatenated to obtain the third detection result vector. Anomaly detection is performed on the slice log based on the third detection result vector to obtain the detection result of the slice log; The step of performing sequence anomaly detection on the segmented log based on the context sequence information of the segmented log to obtain a first detection result vector of the segmented log includes: The context sequence information of the slice log is learned by the Long Short-Term Memory (LSTM) network model, and the slice log is subjected to sequence anomaly detection based on the context sequence information to obtain the first detection result vector of the slice log. The step of performing anomaly detection on the slice log based on the implicit distribution information to obtain a second detection result vector of the slice log includes: The hidden distribution information of the segment log is learned by the variational autoencoder (VAE), and anomaly detection is performed on the segment log based on the feature data of the segment log to obtain the second detection result vector of the segment log. The step of performing anomaly detection on the segment log based on the third detection result vector to obtain the detection result of the segment log includes: The vector information in the third detection result vector is fused using a VAE model to learn the hidden distribution information of the third detection result vector. Anomaly detection is then performed on the slice log based on the hidden distribution information to obtain the detection result of the slice log.
2. The method of claim 1, wherein, The anomaly detection includes at least one of the following: Sequence anomaly detection; Single-point anomaly detection; Quantitative anomaly detection.
3. The method of claim 1, wherein, Before utilizing an ensemble learning strategy for anomaly detection in sliced logs, the method further includes: Perform log preprocessing on the logs to be detected; The preprocessed logs to be detected are parsed to obtain a structured log template with event templates and specific parameters; Feature extraction is performed on the structured log template to obtain at least one slice log corresponding to the log to be detected.
4. The method of claim 1, wherein, The step of performing root cause localization on the segment logs based on the semantic information of the segment logs includes: Each log text information corresponding to the sliced log is input into the text vector model, and the semantic information corresponding to each log text information is extracted. The semantic information forms a text vector. The text vector is input into the semantic model to perform root cause localization on the slice log.
5. The method of claim 4, wherein, Before performing root cause localization on the segment logs based on the semantic information of the segment logs, the method further includes: The text vector model is trained using the training set data, and the semantic information of the log text information in the training set data is extracted to form the text vector of the training set data. A semantic model is trained using the text vectors from the training set data; The trained semantic model is used to determine whether the log expresses positive or negative information, and to identify the root cause of the log behavior when the log expresses negative information.
6. A log processing apparatus characterized by comprising: include: The log anomaly detection module is used to perform anomaly detection on the slice log using an ensemble learning strategy to obtain the detection result of the slice log. The ensemble learning strategy is used to detect various anomalies in the slice log. The log root cause localization module is used to perform root cause localization on the segment log based on the semantic information of the segment log when the detection result indicates that the segment log is abnormal. The log anomaly detection module includes: The first detection unit is used to perform sequence anomaly detection on the slice log based on the context sequence information of the slice log, and obtain a first detection result vector of the slice log; The second detection unit is used to obtain the implicit distribution information of the slice log based on the feature data of the slice log, perform anomaly detection on the slice log based on the implicit distribution information, and obtain a second detection result vector of the slice log; the second detection result is single-point anomaly detection and / or quantitative anomaly detection. An integrated learning unit is used to concatenate the first detection result vector and the second detection result vector to obtain a third detection result vector; and to perform anomaly detection on the slice log based on the third detection result vector to obtain the detection result of the slice log. The first detection unit is specifically used to: learn the context sequence information of the slice log through the Long Short-Term Memory (LSTM) network model, and perform sequence anomaly detection on the slice log based on the context sequence information to obtain the first detection result vector of the slice log; The second detection unit is specifically used to: learn the implicit distribution information of the slice log through a variational autoencoder (VAE), and perform anomaly detection on the slice log based on the feature data of the slice log to obtain a second detection result vector of the slice log; The integrated learning unit is specifically used to: fuse the vector information in the third detection result vector through the VAE model, learn the hidden distribution information of the third detection result vector, perform anomaly detection on the slice log based on the hidden distribution information, and obtain the detection result of the slice log.
7. A log processing device characterized by comprising: Includes processor and transceiver; The processor is configured to: perform anomaly detection on the slice log using an ensemble learning strategy, and obtain the detection result of the slice log, wherein the ensemble learning strategy is used to detect various anomalies in the slice log; If the detection result indicates that the slice log is abnormal, the root cause of the slice log is located based on the semantic information of the slice log. The processor uses an ensemble learning strategy to perform anomaly detection on the slice logs and obtains the detection results of the slice logs, including: Based on the context sequence information of the segmented log, sequence anomaly detection is performed on the segmented log to obtain a first detection result vector of the segmented log; The implicit distribution information of the segmented logs is obtained based on the feature data of the segmented logs. Anomaly detection is performed on the segmented logs based on the implicit distribution information to obtain a second detection result vector of the segmented logs. The second detection result is single-point anomaly detection and / or quantitative anomaly detection. The first detection result vector and the second detection result vector are concatenated to obtain the third detection result vector. Anomaly detection is performed on the slice log based on the third detection result vector to obtain the detection result of the slice log; Optionally, the processor performs sequence anomaly detection on the slice log based on the context sequence information of the slice log to obtain a first detection result vector of the slice log, including: The context sequence information of the slice log is learned by the Long Short-Term Memory (LSTM) network model, and the slice log is subjected to sequence anomaly detection based on the context sequence information to obtain the first detection result vector of the slice log. The processor performs anomaly detection on the slice log based on the implicit distribution information to obtain a second detection result vector of the slice log, including: The hidden distribution information of the segment log is learned by the variational autoencoder (VAE), and anomaly detection is performed on the segment log based on the feature data of the segment log to obtain the second detection result vector of the segment log. The processor performs anomaly detection on the slice log based on the third detection result vector, and obtains the detection result of the slice log including: The vector information in the third detection result vector is fused using the VAE model to learn the hidden distribution information of the third detection result vector. Anomaly detection is performed on the slice log based on the hidden distribution information to obtain the detection result of the slice log.
8. A readable storage medium, on which a program or instructions are stored, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the log processing method as described in any one of claims 1-5.