A log analysis method, device, program product and medium
By generating semantic knowledge bases and deep learning models, the automated analysis of heterogeneous log files is realized, which solves the time-consuming and labor-consuming problem of manual configuration rules in the existing technology, improves the accuracy and efficiency of log analysis, and can automatically identify them. Faulty body.
Patent Information
- Application Number
- CN202411149164.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
The prior art requires manual configuration of rules in log analysis, which consumes energy and time, and limits the application scope of log analysis, so it is impossible to effectively and automatically perform fine-grained analysis of heterogeneous log files.
By obtaining the original log text, generating the data set to be trained based on preset data deduplication rules, using external dictionary and preset embedding models to generate a semantic knowledge base, and combining with deep learning models to train a fine-grained fault log analysis model to realize heterogeneous log files automated analysis.
The fine-grained analysis of heterogeneous log files can be automated without manual configuration rules, which improves the accuracy and efficiency of log analysis, accurately identify faulty subjects, and enhances perception of operating system health.
Smart Images

Figure CN118672990B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of log fault analysis, and in particular to a log parsing method, device, program product and medium. Background Art
[0002] Logs are an important type of data used by various software and hardware to record the operating status of a system. They describe the historical operating status and details of the system. In troubleshooting, security checks, and other work scenarios, log data often needs to be checked and analyzed. By analyzing logs, anomalies can be discovered, errors can be located, or system problems can be eliminated. Currently, log parsing is usually required before log analysis. The process requires manual rule configuration or template extraction using machine learning, code analysis, and other methods, which is both labor-intensive and time-consuming and limits the scope of application of log analysis.
[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a log parsing method, device, program product and medium, which can automatically analyze heterogeneous log files without manually configuring rules, extract and analyze whether a fault occurs in the input log text, and provide the specific subject of the fault. The specific scheme is as follows:
[0005] In a first aspect, the present application discloses a log parsing method, comprising:
[0006] Obtaining original log text, and performing data balancing on the original log text based on preset data deduplication rules to generate a data set to be trained;
[0007] Obtain an external dictionary, and use a preset embedding model to perform semantic embedding using the interpretation text in the external dictionary to generate a semantic knowledge base;
[0008] Using the semantic knowledge base to search the data set to be trained to obtain a data vector corresponding to each word of each log sentence in the data set to be trained, and inputting the data vector into the preset embedding model to generate a corresponding embedding vector;
[0009] The embedded vector is used to train the fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, and the log data generated in the monitoring system environment is parsed by the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0010] Optionally, obtaining the original log text includes:
[0011] Acquire a first original log text that is historically generated in an actual production environment that can produce log data, and / or acquire a second original log text that is publicly available in a network environment.
[0012] Optionally, performing data balancing on the original log text based on a preset data deduplication rule to generate a data set to be trained includes:
[0013] Randomly extracting a preset number of target log texts from the original log texts, and determining remaining log texts from the original log texts according to the target log texts;
[0014] Generate a benchmark data set using the target log text;
[0015] Calculating the similarity between the benchmark data set and the remaining log texts respectively, and determining in turn whether the similarity between the remaining log texts and the benchmark data set exceeds a preset threshold;
[0016] If the similarity between the remaining log text and the benchmark data set exceeds the preset threshold, deleting the remaining log text;
[0017] If the similarity between the remaining log text and the reference data set does not exceed the preset threshold, adding the remaining log text to the reference data set;
[0018] When all the remaining log texts are judged, the current benchmark data set is acquired to obtain the data set to be trained.
[0019] Optionally, the log parsing method further includes:
[0020] Determine a cross entropy loss function used by the fine-grained fault log analysis model to be trained when training the model;
[0021] The corresponding quantity ratios of positive data and negative data in the training data set are counted, and the weights of the positive data and the negative data in the cross entropy loss function are determined based on the quantity ratios; wherein the positive data is the log data running normally in the training data set, and the negative data is the log data running abnormally in the training data set;
[0022] The cross entropy loss function is weighted by using the weight so as to perform model training on the fine-grained fault log analysis model to be trained based on the weighted cross entropy loss function.
[0023] Optionally, the obtaining of an external dictionary and performing semantic embedding using the interpretation text in the external dictionary through a preset embedding model to generate a semantic knowledge base includes:
[0024] Obtaining an external dictionary, and performing data cleaning on the interpretation text in the external dictionary to obtain cleaned interpretation text;
[0025] Inputting the cleaned paraphrase text into a preset embedding model to obtain a corresponding first data vector;
[0026] The cleaned interpretation text is matched one-to-one with the first data vector and stored in a key-value pair format to generate a semantic knowledge base.
[0027] Optionally, the log parsing method further includes:
[0028] When the cleaned interpretation text contains a target interpretation text with multiple meanings, the target interpretation text is input into the preset embedding model to obtain a corresponding second data vector; wherein the second data vector includes data vectors corresponding to multiple interpretations;
[0029] The second data vectors are averaged to determine a target data vector corresponding to the target interpretation text, and the target interpretation text and the target data vector are stored in the semantic knowledge base in the form of key-value pairs.
[0030] Optionally, before the method of using the embedded vector to perform model training on the fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, the method further includes:
[0031] Constructing a context semantic extraction layer, an influence weight calculation module, a graph convolution layer, a mask layer, an attention layer, and a fault classification layer that are connected in sequence to obtain the fine-grained fault log analysis model to be trained;
[0032] Among them, the influence weight calculation module is a module for calculating the relative position weight and the semantic position weight; the relative position weight is used to characterize the distance dependency relationship between words, and the semantic position weight is used to characterize the syntactic dependency information of the sentence.
[0033] Optionally, the using the embedded vector to perform model training on the fine-grained fault log analysis model to be trained includes:
[0034] Based on the context semantic extraction layer, the embedding vector is encoded using a bidirectional long short-term memory model to generate a hidden state vector sequence containing context information;
[0035] Based on the influence weight calculation module, respectively calculating the relative position weight and the semantic position weight, and fusing the relative position weight and the semantic position weight to obtain a fused position weight;
[0036] Based on the graph convolution layer, a double-layer graph convolution neural network is used to perform graph convolution update using the hidden state vector sequence and the fusion position weight to obtain a graph convolution operation result;
[0037] Based on the mask layer, a mask operation is performed on the result of the graph convolution operation to output masked aspect features, and then based on the attention layer, the aspect features are retrieved using an attention mechanism to determine the importance weight of each word;
[0038] Based on the fault classification layer, the hidden state vector sequence is weightedly summed using the importance weight to generate a representation vector for fault classification, and based on the representation vector, the output of the fault classification layer is converted into a probability space containing probabilities corresponding to different log state labels.
[0039] Optionally, calculating the semantic position weight includes:
[0040] Use visualization tools to generate the corresponding syntactic dependency graph for the current log statement;
[0041] The path length between any two words in the log sentence on the syntactic dependency graph is used as the semantic distance between the any two words, so as to calculate the semantic position weight according to the semantic distance.
[0042] Optionally, converting the output of the fault classification layer into a probability space containing probabilities corresponding to different log status labels based on the representation vector includes:
[0043] The representation vector is input into the fully connected layer in the fault analysis layer, and then the output of the fault classification layer is converted into a probability space containing the corresponding probabilities of different log status labels using a target normalization algorithm.
[0044] Optionally, fusing the relative position weight and the semantic position weight to obtain a fused position weight includes:
[0045] The relative position weight and the semantic position weight are fused based on a calculation method of a geometric mean to obtain a fused position weight.
[0046] Optionally, parsing the log data generated in the monitoring system environment by using the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject includes:
[0047] Collecting first log data generated in the system environment to be monitored in real time, and / or collecting second log data generated in the system environment to be monitored according to a preset time period;
[0048] Data preprocessing is performed on the first log data and / or the second log data, and the processed log data is input into the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0049] In a second aspect, the present application discloses an electronic device, comprising:
[0050] Memory for storing computer programs;
[0051] The processor is used to load and execute the computer program to implement the aforementioned log parsing method.
[0052] In a third aspect, the present application discloses a computer program product, including a computer program / instructions, which implement the steps of the aforementioned log parsing method when executed by a processor.
[0053] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned log parsing method when executed by a processor.
[0054] The present application provides a log parsing method, including: obtaining original log text, and performing data balancing on the original log text based on preset data deduplication rules to generate a data set to be trained; obtaining an external dictionary, and performing semantic embedding using the interpretation text in the external dictionary through a preset embedding model to generate a semantic knowledge base; using the semantic knowledge base to search for the data set to be trained to obtain data vectors corresponding to each word of each log sentence in the data set to be trained, and inputting the data vectors into the preset embedding model to generate corresponding embedding vectors; using the embedding vectors to perform model training on a fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, and parsing the log data generated in the monitoring system environment through the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0055] The beneficial technical effects of the present application are as follows: First, in view of the large number and high repetitiveness of the original log texts generated in the actual production environment, preset data deduplication rules are used to improve the problem of data imbalance; secondly, a preset embedding model and an external dictionary are used to generate a semantic knowledge base, and the semantic knowledge base and the preset embedding model are used in combination to process the balanced and optimized training data set, so that the generated embedding representation has richer and more accurate semantic information. Finally, the final fine-grained fault log analysis model is obtained based on the generated embedding vector training, which can be deployed in an operating system environment with log output, and heterogeneous log files can be automatically analyzed in a fine-grained manner without manually configuring rules. The input log data is parsed to analyze whether a fault occurs, and the specific subject of the fault is given to achieve fine-grained log analysis. It further improves the user's perception of the health level of the operating system, and provides users with high-precision, high-accuracy, and high-efficiency fault location services.
[0056] In addition, the present application provides a log analysis device, a program product, and a medium, which correspond to the above-mentioned log analysis method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0058] Figure 1 A schematic diagram of the current log analysis process that combines neural networks and manual parsing technology;
[0059] Figure 2 A flow chart of a log parsing method disclosed in this application;
[0060] Figure 3 A model training flow chart disclosed in this application;
[0061] Figure 4 A fault analysis flow chart disclosed in this application;
[0062] Figure 5 A schematic diagram of the structure of a model disclosed in this application;
[0063] Figure 6 A schematic diagram of relative distances between log statements disclosed in this application;
[0064] Figure 7 A log statement syntactic dependency diagram disclosed in this application;
[0065] Figure 8 A schematic diagram of semantic distance between log statements disclosed in this application;
[0066] Fig. 9 This is a schematic diagram of the structure of a log parsing device disclosed in this application;
[0067] Fig.10 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0069] Since existing logs are usually semi-structured data, log analysis is usually based on log parsing technology, which requires manual analysis of general composition rules, and the use of parsing technology to separate constants and variables, and then analyze them separately to determine the state changes of the system. The accuracy of the analysis depends greatly on the early manual extraction of log rules. Due to the different development specifications of different software and programs, their log formats are usually different. The parsing and analysis of logs in a specific format cannot be directly reused for logs in other formats, so the reusability and versatility of the technology are poor.
[0070] Currently, there is a method of log analysis that combines neural networks with manual analysis technology. The process is as follows:
[0071] 1. Manually parse the logs and extract fields related to the log level;
[0072] 2. Manually count the frequency of logs with the same content;
[0073] 3. Manually count the normal value range of variables involved in the log;
[0074] 4. If Figure 1 As shown in the figure, when the system generates a new log, it first uses the log level to determine whether it is a fault log. For example, a log with the "error" level can be directly determined as a fault log; if it cannot be determined based on the level, the generation frequency and number of times of the content are used to infer whether it is a fault log. For example, if a log is generated multiple times in a short period of time, it may be a fault log; in addition, directly analyze whether the values of the variables involved in the log exceed the normal value range. If so, it may be a fault log.
[0075] 5. When a log is determined to be a fault log in the previous step, the log is input into a text classification convolutional neural network model to extract the information features in the fault log and assign weight values to the information features; wherein the preset neural network model is pre-trained by a preset number of fault logs. Finally, the information features with weight values are sent to the preset machine learning model to classify the fault log into the fault category corresponding to the fault log; wherein the preset machine learning model is pre-trained by the information features of a preset number of fault logs. In this way, the severity and priority of the fault are classified.
[0076] An analysis of the above technical process shows that the premise for determining whether a log is a fault log is to manually extract rules from the log template (for example, specify a field as the log level, define the fault level for different log levels, etc.), and count and define the normal values of certain variables. These operations are all implemented manually, and the accuracy and efficiency are difficult to guarantee, and they are not very universal. In addition, the granularity of the severity analysis of the fault level of this technology is a module in the system, and there is no further analysis of finer-grained sub-modules, so it is impossible to provide more detailed fault information, and the information provided for the subsequent troubleshooting and repair process is not sufficient.
[0077] To this end, the present application provides a log parsing solution that can automatically analyze heterogeneous log files without manually configuring rules, extract and analyze whether a fault occurs based on the input log text, and provide the specific subject of the fault.
[0078] The embodiment of the present invention discloses a log parsing method, see Figure 2 As shown, the method includes:
[0079] Step S11: obtaining the original log text, and performing data balancing on the original log text based on a preset data deduplication rule to generate a data set to be trained.
[0080] In the embodiment of the present application, a fine-grained fault log analysis model combining semantic information and deep learning is proposed to automatically parse heterogeneous log files without manually configuring rules. It can be used in Linux operating systems and other system environments except Linux operating systems. In addition, the heterogeneous log files do not limit the specific rules or file formats of the log text and do not limit the professional field.
[0081] First, in order to make the log analysis model achieve a more accurate effect, it is necessary to use the logs generated in the actual production environment and the log data set publicly available on the network to train the model. Specifically, obtain the first original log text historically generated in the actual production environment that can produce log data, and / or obtain the second original log text publicly available in the network environment.
[0082] It should be pointed out that both the first original log text and the second original log text have data imbalance problems. Since the operating system is relatively stable, there are fewer system crashes and service failures during system operation. Therefore, the fault logs corresponding to the public data set and the logs in actual production account for a small proportion of the total log volume. In other words, in the original log text, there are too few negative data and too many positive data. If the model is trained directly using the original log text with unbalanced samples, it is easy for the model to be insensitive to negative data, affecting the model's recognition accuracy for fault logs.
[0083] Therefore, in the embodiment of the present application, by analyzing the logs generated in the actual production environment, it is found that the positive data (i.e. the log data of normal operation) has the characteristics of large quantity and high repetitiveness. In view of this characteristic, a data balancing optimization layer is introduced, and data deduplication is used to improve the problem of data set imbalance. Specifically, the data balancing of the original log text based on the preset data deduplication rules to generate a training data set includes the following steps:
[0084] Step 1: randomly extracting a preset number of target log texts from the original log texts, and determining the remaining log texts from the original log texts according to the target log texts;
[0085] Step 2: Generate a benchmark data set using the target log text;
[0086] Step 3: Calculate the similarity between the benchmark data set and the remaining log texts respectively, and determine whether the similarity between the remaining log texts and the benchmark data set exceeds a preset threshold;
[0087] Step 4: If the similarity between the remaining log text and the benchmark data set exceeds the preset threshold, the remaining log text is deleted;
[0088] Step 5: If the similarity between the remaining log text and the benchmark data set does not exceed the preset threshold, the remaining log text is added to the benchmark data set;
[0089] Step 6: After all the remaining log texts are judged, the current benchmark data set is obtained to obtain the data set to be trained.
[0090] It can be seen that for the data set corresponding to the original log text, a certain number of logs are first randomly selected as the benchmark data set. For the remaining log data, similarity is calculated with the benchmark data set. When the similarity does not exceed the threshold, it is added to the benchmark data set; if the similarity exceeds the threshold, the log is considered to be a duplicate log and is deleted. When all the remaining log texts are judged, the current benchmark data set is the final training data set.
[0091] In a specific implementation, the similarity between log data can be calculated by cosine similarity. Furthermore, the threshold value for similarity can be artificially set, and its value range is (0-1]. The smaller the value, the lower the requirement for similarity, the smaller the number of final training data screened, but the smaller the possibility of data imbalance, and the better the training effect for the model.
[0092] Step S12: obtaining an external dictionary, and using a preset embedding model to perform semantic embedding using the interpretation text in the external dictionary to generate a semantic knowledge base.
[0093] It is understandable that, as a natural language text, the log cannot be directly read and analyzed by the neural network model, so the log text needs to be converted into a vector that can be recognized and processed by the model. In the embodiment of the present application, a semantic knowledge base is first generated in combination with a preset embedding model, and the semantic knowledge base is used as a retrieval basis for the subsequent input log text, thereby realizing the conversion of the log text into a vector that can be recognized and processed by the model.
[0094] In a specific implementation, the preset embedding model may be BERT (Bidirectional Encoder Representations from Transformers). It uses an attention model as a transformer to establish the relationship between words through an input encoder and an output decoder. Unlike other embedding models that only accept one word input at a time, BERT can take the entire sentence as input, so it can learn the context between words in a sentence well.
[0095] For a given sentence s containing m words, the BERT model automatically inserts a [CLS] symbol at the beginning of the sentence to indicate the beginning of each sentence, and automatically inserts two [SEP] symbols at the beginning and end of the word to indicate the aspect. This means that the order of words in a given sentence is converted to . Then the sentence is input into the BERT model to obtain each word w i The vector representation x i : Among them, BERT (w i ) is the word vector extracted using the pre-trained BERT, dw is the dimension of the word vector. Finally, the sentence s is embedded as X={x1, x2, …, x m}.
[0096] An analysis of the principles of the BERT model shows that the model encodes the input through a multi-layer Transformer encoder, so that the vector can better represent the contextual information, and finally obtains the [CLS] vector. This vector is the weighted average of all words in the sentence text after the attention mechanism, and can better represent the sentence semantics than vectors at other positions. Therefore, the [CLS] vector can represent the lexical interpretation information of the entire sentence. In order to better utilize this feature of the [CLS] vector, in an embodiment of the present application, it is proposed to combine the BERT embedding layer of the semantic knowledge base, introduce the dictionary, an external knowledge base, into the preset embedding model, and improve the accuracy of model recognition.
[0097] Specifically, the process of generating a semantic knowledge base includes the following steps:
[0098] Obtaining an external dictionary, and performing data cleaning on the interpretation text in the external dictionary to obtain cleaned interpretation text;
[0099] Inputting the cleaned paraphrase text into a preset embedding model to obtain a corresponding first data vector;
[0100] The cleaned interpretation text is matched one-to-one with the first data vector and stored in a key-value pair format to generate a semantic knowledge base.
[0101] First, for the selection of external dictionaries, the most common and authoritative English dictionary can be selected. Then the interpretation text in the dictionary is screened and cleaned. After the cleaned interpretation text of a word is input into the BERT model, the [CLS] vector of the text can be obtained. At this time, the vector represents the comprehensive information of the interpretation text of the word, that is, the actual semantic information of the word. Furthermore, after the [CLS] vector of the word and its interpretation is matched one by one, it is stored in the form of a key-value pair to form a semantic knowledge base.
[0102] It should be pointed out that in the process of generating a semantic knowledge base, there may be a situation where a word has multiple meanings. In this case, the mean of the [CLS] vectors of all interpretations is taken as the final semantic vector of the word, and is also stored in the semantic knowledge base in the form of a key-value pair. Specifically, when the cleaned interpretation text contains a target interpretation text with multiple meanings, the target interpretation text is input into the preset embedding model to obtain a corresponding second data vector; wherein the second data vector includes data vectors corresponding to multiple interpretations; the second data vector is averaged to determine the target data vector corresponding to the target interpretation text, and the target interpretation text and the target data vector are stored in the semantic knowledge base in the form of a key-value pair.
[0103] Step S13: using the semantic knowledge base to search the data set to be trained to obtain a data vector corresponding to each word of each log sentence in the data set to be trained, and inputting the data vector into the preset embedding model to generate a corresponding embedding vector.
[0104] In the embodiment of the present application, the semantic knowledge base is used in combination with the BERT embedding layer. Before the log text is subjected to the BERT embedding operation, the [CLS] vector corresponding to each word in the log sentence is first retrieved from the semantic knowledge base. If it exists, the [CLS] vector is used instead of the vocabulary as the initial vector of BERT embedding. In this way, by combining the semantic knowledge base with BERT embedding, the final embedded representation of the log text has richer and more accurate semantic information.
[0105] Step S14: Use the embedded vector to train the fine-grained fault log analysis model to obtain a target fine-grained fault log analysis model, and use the target fine-grained fault log analysis model to parse the log data generated in the monitoring system environment to determine the fault log and the corresponding fault subject.
[0106] In the embodiment of the present application, after the embedding vector of the log is obtained, the fine-grained fault log analysis model to be trained is trained based on the embedding vector. Figure 3 The figure shows the flow chart of the model during training. During the training process, the interpretation text of the external dictionary is cleaned and input into BERT for semantic embedding to obtain a semantic knowledge base; the training data set is cleaned and balanced and optimized, and the semantic knowledge base is used to retrieve the data set obtained after balanced optimization to obtain the corresponding data vector, which is used to input into the BERT model to obtain the corresponding embedding representation. Then the fine-grained fault log analysis model to be trained uses the embedding representation generated by the data set for model training. The fine-grained fault log analysis model is tuned to determine whether its accuracy meets the requirements to determine the final target fine-grained fault log analysis model.
[0107] In the embodiment of the present application, after the model training is completed, it is deployed to the system environment to be monitored. The new log data generated by the system can be automatically parsed to analyze whether a fault occurs, and the specific subject of the fault (such as a component of a service) is given to achieve fine-grained log analysis.
[0108] Specifically, when the log data generated in the monitoring system environment is parsed by the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject, the following steps are specifically included:
[0109] Collecting first log data generated in the system environment to be monitored in real time, and / or collecting second log data generated in the system environment to be monitored according to a preset time period;
[0110] Data preprocessing is performed on the first log data and / or the second log data, and the processed log data is input into the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0111] It can be seen that logs can be collected and analyzed in real time according to needs; logs can also be collected and analyzed uniformly according to a certain time period. In addition, the collection scope of logs can also be freely set according to needs, not limited to a certain service or format. After automatically collecting text content, the data is preprocessed and then input into the model. For example, numbers and symbols can be cleaned and then input into the model for analysis, and finally the specific subject of the fault is given, which further improves the user's perception of the health level of the operating system and provides users with high-precision, high-accuracy, and high-efficiency fault location services.
[0112] like Figure 4The diagram shown is a schematic diagram of using a target fine-grained fault log analysis model to analyze the log data generated in the monitoring system environment and analyze whether a fault occurs. New log data is generated for the system environment currently deployed by the model and used as the model input. First, the log text is cleaned, and then the log text is mapped to a low-dimensional vector space to achieve conversion into a vector that can be recognized and processed by the model, and input into the target fine-grained fault log analysis model. The model can determine whether the current log is a fault log by analyzing the vector. If it is a fault log, the fault subject that caused the fault log to be generated will also be given. In a feasible implementation, the model can also give the severity of the fault log after analyzing the vector, as well as the fault severity of the fault subject or submodule involved in the log. A corresponding fault level can be set for the fault severity, and by comparing and judging the fault level, an early warning message can be issued according to the fault severity, and a fault warning is given to the user before the system crashes or fails. Further, the staff can formulate and adopt corresponding strategies to process the fault log according to the early warning information.
[0113] The present application provides a log parsing method, including: obtaining original log text, and performing data balancing on the original log text based on preset data deduplication rules to generate a data set to be trained; obtaining an external dictionary, and performing semantic embedding using the interpretation text in the external dictionary through a preset embedding model to generate a semantic knowledge base; using the semantic knowledge base to search for the data set to be trained to obtain data vectors corresponding to each word of each log sentence in the data set to be trained, and inputting the data vectors into the preset embedding model to generate corresponding embedding vectors; using the embedding vectors to perform model training on a fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, and parsing the log data generated in the monitoring system environment through the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0114] The beneficial technical effects of the present application are as follows: First, in view of the large number and high repetitiveness of the original log texts generated in the actual production environment, preset data deduplication rules are used to improve the problem of data imbalance; secondly, a preset embedding model and an external dictionary are used to generate a semantic knowledge base, and the semantic knowledge base and the preset embedding model are used in combination to process the balanced and optimized training data set, so that the generated embedding representation has richer and more accurate semantic information. Finally, the final fine-grained fault log analysis model is obtained based on the generated embedding vector training, which can be deployed in an operating system environment with log output, and heterogeneous log files can be automatically analyzed in a fine-grained manner without manually configuring rules. The input log data is parsed to analyze whether a fault occurs, and the specific subject of the fault is given to achieve fine-grained log analysis. It further improves the user's perception of the health level of the operating system, and provides users with high-precision, high-accuracy, and high-efficiency fault location services.
[0115] Based on the above embodiment, this embodiment will specifically explain step S14 in the above embodiment. Before specifically explaining step S14, the structure of the target fine-grained fault log analysis model is first explained. Figure 5 The figure shows an exemplary schematic diagram of a hierarchical structure of a target fine-grained fault log analysis model. Among them, the original log text will be subjected to data balancing through the data balancing optimization layer disclosed in the aforementioned embodiment, and then the BERT embedding layer combined with the semantic knowledge base will generate an embedding representation corresponding to the corresponding data set to be trained. Among them, for more specific working processes of the data balancing optimization layer and the BERT embedding layer combined with the semantic knowledge base, please refer to the corresponding content disclosed in the aforementioned embodiment, which will not be repeated here. Furthermore, for the fine-grained fault log analysis model to be trained, it is sequentially connected and constructed based on the context semantic extraction layer, influence weight calculation module, graph convolution layer, mask layer, attention layer, and fault classification layer.
[0116] Next, the process of model training is described in detail. The process of using the embedding vector to train the fine-grained fault log analysis model to be trained may include the following steps:
[0117] Step 1: Based on the contextual semantic extraction layer, the embedding vector is encoded using a bidirectional long short-term memory model to generate a hidden state vector sequence containing contextual information.
[0118] In the embodiment of the present application, after obtaining the embedded representation of the log, the bidirectional long short-term memory model (Bidirectional Long Short-Term Memory, BiLSTM) is used to encode these embedded vectors, aiming to capture the forward and reverse contextual dependencies in the log sequence.
[0119] BiLSTM integrates historical and future contextual information into the hidden state of the current time step through forward and back propagation algorithms, generating a hidden state vector sequence H containing comprehensive contextual information. c , which contains the hidden state vector of the bidirectional long short-term memory model for all time steps.
[0120] Step 2: Based on the influence weight calculation module, the relative position weight and the semantic position weight are calculated respectively, and the relative position weight and the semantic position weight are fused to obtain a fused position weight.
[0121] In the present application embodiment, after obtaining H c After that, a two-layer graph convolutional neural network (GCN) is further used to update the syntactic dependency information in the sentence and the long-distance dependency relationship between words. To this end, the concept of syntactic dependency graph is introduced, and the connection relationship between nodes (i.e., words) in the graph is represented by the unweighted graph adjacency matrix A. In the adjacency matrix A, for two nodes i and j with a dependency relationship, set A ij =A ji =1; at the same time, in order to retain the node's self-information, for each node i, set A ii = 1. Through GCN, H c The joint processing of and A can extract the deep syntactic relationship between words and phrases, thereby enhancing the accuracy of semantic understanding of log data.
[0122] It is understandable that log text is usually a brief description of the event around the event subject (service, component, etc.), which conforms to the semantic rules and grammatical rules of natural language, and the descriptive words at different positions in the text have different influences on the event subject. In the embodiment of the present application, firstly, based on the influence weight calculation module, the influence of the descriptive words is judged from the two dimensions of semantics and relative position, and the comprehensive influence weight of the descriptive words is calculated. Among them, the influence weight calculation module is a module that calculates the relative position weight and the semantic position weight; the relative position weight is used to characterize the distance dependency relationship between words, and the semantic position weight is used to characterize the syntactic dependency information of the sentence.
[0123] In a specific implementation, for the event subject, the influence of the descriptive word is measured by the relative position distance between other descriptive words in the same sentence and the subject. For example, for a log of the Grafana component "Grafana-server failed to look up user based on cookie.", the aspect word in the sentence is "Grafana-server", which is the event subject in the event description; the relative position distance of other descriptive words in the log is as follows: Figure 6 When calculating the relative distance between a descriptor and the event subject, the closest main component of the descriptor is used as the calculation basis. For example, the distance between the descriptor "failed" and the event subject "Grafana-server" is calculated according to the distance between it and the component "server", so the relative distance is 1. In a similar way, the relative distances between all descriptor components and the event subject can be calculated: "to look up user based on cookie." The relative distances from "server" are 2, 3, 4, 5, 6, 7, and 8 respectively.
[0124] Further, after the sentence disclosed in the above embodiment is input into the BERT model, each word w i The vector representation x i Formula , can be represented by the descriptor w i The relative position distance between the event subject and w is calculated i The relative position weight q i :
[0125] ;
[0126] The subscripts from τ to τ+m are the subscripts corresponding to an event body of length m+1, and n is the maximum subscript.
[0127] It should be pointed out that when using relative distance weights for subsequent position-aware conversion, a common situation will be ignored: in long sentences with complex structures, although some descriptive words are relatively far away from the event subject, they have a significant impact on the event subject at the semantic level. If this situation is ignored by the conversion method that relies solely on relative distance weights, it may lead to a one-sided understanding of the description of the event subject. Therefore, in order to capture this semantic level influence more comprehensively and accurately, in another specific implementation method, semantic position weights are introduced for comprehensive consideration. The importance of semantic position weights lies in that they can make up for the shortcomings of relative distance weights in semantic understanding, especially when dealing with complex long sentences. By combining semantic position weights with relative distance weights, the true influence of descriptive words on the event subject in a sentence can be more accurately reflected. This combination makes log analysis more comprehensive and closer to the complexity and diversity of natural language.
[0128] Specifically, calculating the semantic position weight includes the following steps: using a visualization tool to generate a corresponding syntactic dependency graph for the current log statement; taking the path length of any two words in the log statement on the syntactic dependency graph as the semantic distance between the any two words, so as to calculate the semantic position weight based on the semantic distance.
[0129] It is understandable that the syntactic dependency graph can reveal the syntactic relationship between the components in the sentence, thereby helping to determine the semantic connection between the descriptor and the event subject. Through the analysis of the syntactic dependency graph, the semantic position weight of the descriptor relative to the event subject can be accurately calculated.
[0130] For example, for a log of the Grafana component "Grafana-server failed to look upuser based on cookie.", first use the visualization tool to generate a syntactic dependency graph such as Figure 7 As shown, the arrow relationship between each word corresponds to the relevant noun explanation of syntactic analysis, such as combination represents "compound", noun subject represents "nsubj", clause complement represents "xcomp", non-main verbs and auxiliary words represent "aux", etc.
[0131] After obtaining the syntactic dependency graph, the path length between the two words on the syntactic dependency graph is used as the semantic distance between the two words, such as Figure 8The following is a schematic diagram of the semantic distance corresponding to the above log text. Among them, the serial numbers (1) to (10) respectively correspond to the position order of each word in "Grafana-server failed to look up user based on cookie.", that is, (1) corresponds to the word Grafana-, (2) corresponds to the word server, and so on. According to the semantic distance calculation, the semantic position weight of w i is as follows:
[0132] ;
[0133] where d max is the maximum value of the path length in the syntactic dependency graph, and d i is the path length (i.e., the number of hops) between the descriptor w i and the aspect word in the syntactic dependency graph. When the number of hops between the descriptor and the aspect word is smaller (i.e., the semantic distance is smaller), the influence of this word on the semantics of the descriptor is stronger, and its semantic position weight value is larger. The finally calculated s i is the semantic position weight of the descriptor w i .
[0134] Furthermore, after calculating the semantic position weight s i using the semantic distance, it is fused with the relative position weight q i as the final fused position weight pos i of the descriptor w i . This weight not only comprehensively considers the semantic proximity and physical position relationship between the descriptor and the event subject, but also can more comprehensively reflect the actual influence and importance of the descriptor on the event subject in the sentence.
[0135] To better balance the two, a geometric mean calculation method is used to calculate the fused weight. Specifically, based on the geometric mean calculation method, the relative position weight and the semantic position weight are fused to obtain the fused position weight. This method ensures that the two weights are evenly considered in the fusion process, avoiding biases caused by too large or too small weights in a single dimension. Among them, .
[0136] Step 3: Based on the graph convolutional layer, use the hidden state vector sequence and the fused position weight through a two-layer graph convolutional neural network for graph convolutional update to obtain the graph convolutional operation result.
[0137] After obtaining the dependency adjacency matrix A and the fused position weight pos i (0 < i < n + 1), use them as important parameters for the subsequent graph convolutional layer to perform graph convolutional update with position-aware transformation:
[0138] ; A ij For word w i With the word w j The adjacency matrix value corresponding to the dependency relationship, the weight parameter W l is a trainable vector, is the result of the position-aware transformation of the output of the previous layer of graph convolutional neural network for word node i, and finally is the original node representation of node i in layer l.
[0139] pos i To fuse the position weight, the meaning of this formula is the output of GCN Perform location-aware transformations.
[0140] ;d i is the degree of word node i in the syntactic dependency graph.
[0141] . Bias b l is a trainable parameter.
[0142] Get the original node representation After that, the activation function conversion operation is performed on it to obtain the final graph convolution operation result , that is, the state vector of the i-th word node on the l-th layer of the graph convolutional neural network.
[0143] It should be noted that the input of the initial graph convolutional neural network is the output of the contextual semantic extraction layer , that is, H 0 =H c The purpose of this operation is to make the initial node information have contextual semantic information. The final output of the L-layer graph convolutional network is After actual testing, the model achieves the best effect when the number of layers L is 2, so the default value of L in this embodiment can be 2.
[0144] Step 4: Based on the mask layer, a mask operation is performed on the result of the graph convolution operation to output masked aspect features, and then based on the attention layer, the aspect features are retrieved using the attention mechanism to determine the importance weight of each word.
[0145] After completing the graph convolution operation, in order to emphasize the significant features of aspect words, a mask operation is performed on the generated vector. Then, the masked aspect-related features are retrieved using the attention mechanism. This step aims to identify words or phrases that have a significant impact on the sentiment polarity of the target word and calculate the importance weight of each context word.
[0146] Step 5: Based on the fault classification layer, the importance weights are used to perform weighted summation on the hidden state vector sequence to generate a representation vector for fault classification, and based on the representation vector, the output of the fault classification layer is converted into a probability space containing the corresponding probabilities of different log state labels.
[0147] Then, these importance weights are combined with the vector H output by the context semantic extraction layer c Combined, these importance weights are used to c The vectors in are weighted summed to generate a final vector r for fault classification. In the fault analysis layer, the vector r is used for fault identification and output, and parameter update and optimization training are performed simultaneously. Specifically, the representation vector is input into the fully connected layer in the fault analysis layer, and then the target normalization algorithm is used to convert the output of the fault classification layer into a probability space containing the corresponding probabilities of different log status labels. Exemplarily, once the final representation vector is obtained, it is first input into the fully connected layer, and then the softmax normalization algorithm can be applied to convert the output into a probability space containing the corresponding probabilities of the "fault" and "normal" labels.
[0148] Based on the above embodiment, in a feasible implementation, in order to further improve the model training effect, the cross entropy loss function in the model training process is improved in the embodiment of the present application. Specifically, the following steps may also be included:
[0149] Step 1: Determine the cross entropy loss function used by the fine-grained fault log analysis model to be trained when training the model;
[0150] Step 2: Count the corresponding quantity proportions of positive data and negative data in the training data set, and determine the weights of the positive data and the negative data in the cross entropy loss function based on the quantity proportions; wherein the positive data is the log data running normally in the training data set, and the negative data is the log data running abnormally in the training data set;
[0151] Step three: Use the weight to weight the cross entropy loss function so as to perform model training on the fine-grained fault log analysis model to be trained based on the weighted cross entropy loss function.
[0152] In the embodiment of the present application, the proportion of positive data and negative data in the training data set after data balance optimization is counted, and the weights of the two parts of data in the cross entropy loss function are calculated according to their proportions:
[0153] ;
[0154] Among them, N is the total number of samples, N n is the number of negative samples, N p is the number of positive samples, x i Corresponding to each sample. In order to make the model pay more attention to the learning of negative samples and emphasize the importance of negative samples during training, it is necessary to increase the loss weight of negative samples and reduce the loss weight of positive samples. p >N n , take the proportion of positive samples in the total number of samples as the loss weight of negative samples (larger), take the proportion of negative samples in the total number of samples as the loss weight of positive samples (smaller).
[0155] In the above formula, when y i =1, the first item " " is effective and is used to calculate the loss of positive samples; when y i =0, the second term " " is effective and is used to calculate the loss of negative samples. After actual testing, the training effect is improved when the above weighted cross entropy loss function is used during training, and the model's recognition of negative samples (fault logs) is improved.
[0156] It can be seen that by improving the cross entropy loss function in the model training process, the model's sensitivity to negative data, which accounts for a relatively small proportion, is improved. Even if the original training data still has data imbalance after passing through the data balancing optimization layer, the impact of data imbalance on the model training effect can be minimized.
[0157] Correspondingly, the present application also discloses a log parsing device, see Fig. 9 As shown, the device comprises:
[0158] The data set generation module 11 is used to obtain the original log text and perform data balancing on the original log text based on a preset data deduplication rule to generate a data set to be trained;
[0159] A semantic knowledge base generation module 12 is used to obtain an external dictionary and perform semantic embedding using the interpretation text in the external dictionary through a preset embedding model to generate a semantic knowledge base;
[0160] The embedding vector generating module 13 is used to search the data set to be trained by using the semantic knowledge base to obtain the data vector corresponding to each word of each log sentence in the data set to be trained, and input the data vector into the preset embedding model to generate the corresponding embedding vector;
[0161] A model training module 14 is used to perform model training on the fine-grained fault log analysis model to be trained using the embedded vector to obtain a target fine-grained fault log analysis model;
[0162] The log parsing module 15 is used to parse the log data generated in the monitoring system environment through the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0163] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0164] It can be seen that the above scheme of this embodiment includes: obtaining the original log text, and performing data balancing on the original log text based on the preset data deduplication rules to generate a data set to be trained; obtaining an external dictionary, and performing semantic embedding using the interpretation text in the external dictionary through a preset embedding model to generate a semantic knowledge base; using the semantic knowledge base to search the data set to be trained to obtain the data vector corresponding to each word of each log sentence in the data set to be trained, and inputting the data vector into the preset embedding model to generate a corresponding embedding vector; using the embedding vector to perform model training on the fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, and using the target fine-grained fault log analysis model to parse the log data generated in the monitoring system environment to determine the fault log and the corresponding fault subject.
[0165] The beneficial technical effects of the present application are as follows: First, in view of the large number and high repetitiveness of the original log texts generated in the actual production environment, preset data deduplication rules are used to improve the problem of data imbalance; secondly, a preset embedding model and an external dictionary are used to generate a semantic knowledge base, and the semantic knowledge base and the preset embedding model are used in combination to process the balanced and optimized training data set, so that the generated embedding representation has richer and more accurate semantic information. Finally, the final fine-grained fault log analysis model is obtained based on the generated embedding vector training, which can be deployed in an operating system environment with log output, and heterogeneous log files can be automatically analyzed in a fine-grained manner without manually configuring rules. The input log data is parsed to analyze whether a fault occurs, and the specific subject of the fault is given to achieve fine-grained log analysis. It further improves the user's perception of the health level of the operating system, and provides users with high-precision, high-accuracy, and high-efficiency fault location services.
[0166] In a specific implementation, the data set generation module 11 includes:
[0167] The log text acquisition module is used to acquire a first original log text historically generated in an actual production environment capable of producing log data, and / or to acquire a second original log text publicly available in a network environment.
[0168] A reference data set generation module, used to randomly extract a preset number of target log texts from the original log texts, and determine the remaining log texts from the original log texts according to the target log texts; and generate a reference data set using the target log texts;
[0169] The similarity calculation module is used to calculate the similarity between the benchmark data set and the remaining log text respectively, and judge in turn whether the similarity between the remaining log text and the benchmark data set exceeds a preset threshold; if the similarity between the remaining log text and the benchmark data set exceeds the preset threshold, the remaining log text is deleted; if the similarity between the remaining log text and the benchmark data set does not exceed the preset threshold, the remaining log text is added to the benchmark data set; when the remaining log text is judged, the current benchmark data set is obtained to obtain the data set to be trained.
[0170] In a specific implementation, the log parsing device further includes a cross entropy function optimization module, which is used to:
[0171] Determine a cross entropy loss function used by the fine-grained fault log analysis model to be trained when training the model;
[0172] The corresponding quantity ratios of positive data and negative data in the training data set are counted, and the weights of the positive data and the negative data in the cross entropy loss function are determined based on the quantity ratios; wherein the positive data is the log data running normally in the training data set, and the negative data is the log data running abnormally in the training data set;
[0173] The cross entropy loss function is weighted by using the weight so as to perform model training on the fine-grained fault log analysis model to be trained based on the weighted cross entropy loss function.
[0174] In a specific implementation, the semantic knowledge base generation module 12 includes:
[0175] A data cleaning module is used to obtain an external dictionary and perform data cleaning on the interpretation text in the external dictionary to obtain the cleaned interpretation text;
[0176] The first storage module is used to input the cleaned interpretation text into a preset embedding model to obtain a corresponding first data vector; the cleaned interpretation text is matched with the first data vector one by one and stored in a key-value pair format to generate a semantic knowledge base.
[0177] The second storage module is used for inputting the target interpretation text into the preset embedding model to obtain a corresponding second data vector when the cleaned interpretation text contains a target interpretation text with multiple meanings; wherein the second data vector includes data vectors corresponding to multiple interpretations; taking the average of the second data vector to determine the target data vector corresponding to the target interpretation text, and storing the target interpretation text and the target data vector in the form of key-value pairs in the semantic knowledge base.
[0178] In a specific implementation, the model training module 14 includes:
[0179] A model construction module, used to construct a context semantic extraction layer, an influence weight calculation module, a graph convolution layer, a mask layer, an attention layer, and a fault classification layer that are connected in sequence to obtain the fine-grained fault log analysis model to be trained;
[0180] Among them, the influence weight calculation module is a module for calculating the relative position weight and the semantic position weight; the relative position weight is used to characterize the distance dependency relationship between words, and the semantic position weight is used to characterize the syntactic dependency information of the sentence.
[0181] The syntactic dependency graph processing module is used to generate a corresponding syntactic dependency graph for the current log statement using a visualization tool; the path length between any two words in the log statement on the syntactic dependency graph is used as the semantic distance between the any two words, so as to calculate the semantic position weight according to the semantic distance.
[0182] The probability space conversion module is used to input the representation vector into the fully connected layer in the fault analysis layer, and then use the target normalization algorithm to convert the output of the fault classification layer into a probability space containing the corresponding probabilities of different log status labels.
[0183] The fused position weight determination module is used to fuse the relative position weight and the semantic position weight based on the calculation method of the geometric mean to obtain the fused position weight.
[0184] In a specific implementation, the model training module 14 is specifically used to:
[0185] Based on the context semantic extraction layer, the embedding vector is encoded using a bidirectional long short-term memory model to generate a hidden state vector sequence containing context information;
[0186] Based on the influence weight calculation module, respectively calculating the relative position weight and the semantic position weight, and fusing the relative position weight and the semantic position weight to obtain a fused position weight;
[0187] Based on the graph convolution layer, a double-layer graph convolution neural network is used to perform graph convolution update using the hidden state vector sequence and the fusion position weight to obtain a graph convolution operation result;
[0188] Based on the mask layer, a mask operation is performed on the result of the graph convolution operation to output masked aspect features, and then based on the attention layer, the aspect features are retrieved using an attention mechanism to determine the importance weight of each word;
[0189] Based on the fault classification layer, the hidden state vector sequence is weightedly summed using the importance weight to generate a representation vector for fault classification, and based on the representation vector, the output of the fault classification layer is converted into a probability space containing probabilities corresponding to different log state labels.
[0190] In a specific implementation, the log parsing module 15 is specifically used to:
[0191] Collecting first log data generated in the system environment to be monitored in real time, and / or collecting second log data generated in the system environment to be monitored according to a preset time period;
[0192] Data preprocessing is performed on the first log data and / or the second log data, and the processed log data is input into the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
[0193] Furthermore, the present application also discloses an electronic device. Fig.10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the scope of use of the present application.
[0194] Fig.10 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the log parsing method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be a computer.
[0195] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0196] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The data 223 may include various data. The storage method may be temporary storage or permanent storage.
[0197] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the log parsing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0198] Furthermore, the embodiments of the present application also disclose a computer-readable storage medium, wherein the computer-readable storage medium mentioned here includes a random access memory (Random Access Memory, RAM), a memory, a read-only memory (Read-Only Memory, ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. Wherein, the computer program implements the aforementioned log parsing method when executed by the processor. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0199] Furthermore, an embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements any one of the above-mentioned log parsing methods when executed by a processor.
[0200] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0201] The steps of the log parsing method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0202] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0203] The above is a detailed introduction to a log parsing method, device, program product and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A log parsing method, characterized in that: include: Obtaining original log text, and performing data balancing on the original log text based on preset data deduplication rules to generate a data set to be trained; The preset data deduplication rule uses data deduplication to improve the imbalance of data sets by introducing a data balancing optimization layer; Obtain an external dictionary, and use a preset embedding model to perform semantic embedding using the interpretation text in the external dictionary to generate a semantic knowledge base; Using the semantic knowledge base to search the data set to be trained to obtain a data vector corresponding to each word of each log sentence in the data set to be trained, and inputting the data vector into the preset embedding model to generate a corresponding embedding vector; The embedded vector is used to train the fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, and the log data generated in the monitoring system environment is parsed by the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
2. The log parsing method according to claim 1, characterized in that: The obtaining of the original log text includes: Acquire a first original log text that is historically generated in an actual production environment that can produce log data, and / or acquire a second original log text that is publicly available in a network environment.
3. The log parsing method according to claim 1, characterized in that: The performing data balancing on the original log text based on the preset data deduplication rule to generate a data set to be trained includes: Randomly extracting a preset number of target log texts from the original log texts, and determining remaining log texts from the original log texts according to the target log texts; Generate a benchmark data set using the target log text; Calculating the similarity between the benchmark data set and the remaining log texts respectively, and determining in turn whether the similarity between the remaining log texts and the benchmark data set exceeds a preset threshold; If the similarity between the remaining log text and the benchmark data set exceeds the preset threshold, deleting the remaining log text; If the similarity between the remaining log text and the reference data set does not exceed the preset threshold, adding the remaining log text to the reference data set; When all the remaining log texts are judged, the current benchmark data set is acquired to obtain the data set to be trained.
4. The log parsing method according to claim 1, characterized in that: Also includes: Determine a cross entropy loss function used by the fine-grained fault log analysis model to be trained when training the model; The corresponding quantity ratios of positive data and negative data in the training data set are counted, and the weights of the positive data and the negative data in the cross entropy loss function are determined based on the quantity ratios; wherein the positive data is the log data running normally in the training data set, and the negative data is the log data running abnormally in the training data set; The cross entropy loss function is weighted by using the weight so as to perform model training on the fine-grained fault log analysis model to be trained based on the weighted cross entropy loss function.
5. The log parsing method according to claim 1, characterized in that: The obtaining of the external dictionary and performing semantic embedding using the interpretation text in the external dictionary through a preset embedding model to generate a semantic knowledge base includes: Obtaining an external dictionary, and performing data cleaning on the interpretation text in the external dictionary to obtain cleaned interpretation text; Inputting the cleaned paraphrase text into a preset embedding model to obtain a corresponding first data vector; The cleaned interpretation text is matched one-to-one with the first data vector and stored in a key-value pair format to generate a semantic knowledge base.
6. The log parsing method according to claim 5, characterized in that: Also includes: When the cleaned interpretation text contains a target interpretation text with multiple meanings, the target interpretation text is input into the preset embedding model to obtain a corresponding second data vector; wherein the second data vector includes data vectors corresponding to multiple interpretations; The second data vectors are averaged to determine a target data vector corresponding to the target interpretation text, and the target interpretation text and the target data vector are stored in the semantic knowledge base in the form of key-value pairs.
7. The log parsing method according to claim 1, characterized in that: Before the method uses the embedding vector to perform model training on the fine-grained fault log analysis model to be trained to obtain a target fine-grained fault log analysis model, the method further includes: Constructing a context semantic extraction layer, an influence weight calculation module, a graph convolution layer, a mask layer, an attention layer, and a fault classification layer that are connected in sequence to obtain the fine-grained fault log analysis model to be trained; Among them, the influence weight calculation module is a module for calculating the relative position weight and the semantic position weight; the relative position weight is used to characterize the distance dependency relationship between words, and the semantic position weight is used to characterize the syntactic dependency information of the sentence.
8. The log parsing method according to claim 7, characterized in that: The using the embedded vector to perform model training on the fine-grained fault log analysis model to be trained includes: Based on the context semantic extraction layer, the embedding vector is encoded using a bidirectional long short-term memory model to generate a hidden state vector sequence containing context information; Based on the influence weight calculation module, respectively calculating the relative position weight and the semantic position weight, and fusing the relative position weight and the semantic position weight to obtain a fused position weight; Based on the graph convolution layer, a double-layer graph convolution neural network is used to perform graph convolution update using the hidden state vector sequence and the fusion position weight to obtain a graph convolution operation result; Based on the mask layer, a mask operation is performed on the result of the graph convolution operation to output masked aspect features, and then based on the attention layer, the aspect features are retrieved using an attention mechanism to determine the importance weight of each word; Based on the fault classification layer, the hidden state vector sequence is weightedly summed using the importance weight to generate a representation vector for fault classification, and based on the representation vector, the output of the fault classification layer is converted into a probability space containing probabilities corresponding to different log state labels.
9. The log parsing method according to claim 8, characterized in that: Calculating the semantic position weight includes: Use visualization tools to generate the corresponding syntactic dependency graph for the current log statement; The path length between any two words in the log sentence on the syntactic dependency graph is used as the semantic distance between the any two words, so as to calculate the semantic position weight according to the semantic distance.
10. The log parsing method according to claim 8, characterized in that: The step of converting the output of the fault classification layer into a probability space containing probabilities corresponding to different log status labels based on the representation vector includes: The representation vector is input into the fully connected layer in the fault analysis layer, and then the target normalization algorithm is used to convert the output of the fault classification layer into a probability space containing the corresponding probabilities of different log status labels.
11. The log parsing method according to claim 8, characterized in that: The fusing the relative position weight and the semantic position weight to obtain a fused position weight includes: The relative position weight and the semantic position weight are fused based on a calculation method of a geometric mean to obtain a fused position weight.
12. The log parsing method according to any one of claims 1 to 11, characterized in that: The target fine-grained fault log analysis model is used to analyze the log data generated in the monitoring system environment to determine the fault log and the corresponding fault subject, including: Collecting first log data generated in the system environment to be monitored in real time, and / or collecting second log data generated in the system environment to be monitored according to a preset time period; Data preprocessing is performed on the first log data and / or the second log data, and the processed log data is input into the target fine-grained fault log analysis model to determine the fault log and the corresponding fault subject.
13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to load and execute the computer program to implement the log parsing method according to any one of claims 1 to 12.
14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the log parsing method according to any one of claims 1 to 12 are implemented.
15. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein the computer program, when executed by a processor, implements the log parsing method according to any one of claims 1 to 12.
