A method, device, equipment and storage medium for constructing a log data event graph

Abstract

The present invention provides a method, apparatus, device, and storage medium for constructing a log data event graph. The method comprises: extracting time information corresponding to each log event in received log data; extracting trigger words corresponding to each log event based on log statements corresponding to each log event in the log data; classifying the log events in the log data based on the trigger words corresponding to each log event, dividing log events containing the same trigger word into a log event set, and extracting arguments corresponding to each log event; constructing a log event chain for each log event in the log data based on the time information corresponding to each log event in the log data; and constructing an event graph for the log data based on the log event chain and the time information, trigger words, and arguments corresponding to each log event. The log event graph created by the present invention characterizes the correlation between log events and improves the accuracy and efficiency of data queries.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for constructing a log data event graph. Background Art

[0002] With the rapid development of computer and internet technologies, the rapid growth of data has led to an increasing amount of information, and people are demanding higher and higher quality and accuracy in information queries. Logging systems are a crucial component of operations and maintenance, recording all actions taken by the system and operators, and recording these actions according to the developer's specifications.

[0003] Log data is typically queried manually, which is time-consuming and lacks accuracy. Other approaches rely solely on simple log knowledge graphs. However, knowledge graphs, with their isolated knowledge points, are unable to represent the complex logic contained within log data. Therefore, this approach also struggles to guarantee data query accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and storage medium for constructing a log data event graph to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present invention provides a method for constructing a log data event graph, the method comprising the following steps:

[0006] The first line of the received log data that conforms to the time format is treated as a log event, and the time information corresponding to each log event is extracted;

[0007] Extracting trigger words corresponding to each log event based on the log statements corresponding to each log event in the log data;

[0008] Classifying the log events in the log data according to the trigger words corresponding to the respective log events, dividing the log events containing the same trigger words into a log event set, and extracting arguments corresponding to the respective log events based on the respective log event sets;

[0009] Constructing a log event chain for each log event in the log data according to the time information corresponding to each log event in the log data;

[0010] An event graph of the log data is constructed based on the log event chain and the time information, trigger words, and arguments corresponding to each log event.

[0011] In some embodiments of the present invention, extracting the trigger word corresponding to each log event includes:

[0012] Segmenting the log statements of each log event in the log data to obtain a segmentation set corresponding to each log event;

[0013] Counting the word frequency inverse file frequency of each word in the word segmentation set of each log event in the log data;

[0014] Sort the words in the word set of each log event according to the word frequency inverse file frequency, cluster the sorted word set of each log event, and obtain a trigger word list corresponding to each log event;

[0015] The log data is annotated with keywords based on the trigger word list, and a trigger word prediction model is trained using the annotated log data. The trigger word prediction model is used to predict the trigger word of each log event in the log data.

[0016] In some embodiments of the present invention, before segmenting the log statement of each log event in the log data, the method further includes:

[0017] Segment each log event into sentences based on specified characters, and delete characters that have no semantic meaning in each segmented sentence to obtain the filtered log statement corresponding to each log event;

[0018] Perform word segmentation on the filter log statement corresponding to each log event.

[0019] In some embodiments of the present invention, predicting the trigger word of each log event in the log data using the trigger word prediction model includes:

[0020] Using the trigger word prediction model to predict candidate trigger words and scores corresponding to the candidate trigger words for each log event in the log data, and using the candidate trigger word with the highest score as the trigger word for the corresponding log event;

[0021] If the scores of the candidate trigger words of the log event predicted by the trigger word prediction model are all less than the preset scores, the word frequency inverse file frequencies of the candidate trigger words of the log event in the log data are counted, and the candidate trigger words are sorted and clustered based on the word frequency inverse file frequencies to obtain a new trigger word list for each log event. The trigger words of each log event are re-predicted based on the new trigger word list and the trigger word prediction model until the score of the trigger word of each log event is greater than or equal to the preset score.

[0022] In some embodiments of the present invention, extracting arguments corresponding to each log event includes:

[0023] The log statements of the log events in each log event set are input into the clustering model respectively, the log events in each log event set are semantically clustered, and the semantic vector of each cluster center cluster is used as the argument corresponding to each log event in the corresponding log event set.

[0024] In some embodiments of the present invention, constructing a log event chain for each log event in the log data according to time information corresponding to each log event in the log data includes:

[0025] Sort the log events in the log data according to the time information corresponding to each log event in the log data to obtain a log event set;

[0026] Segmenting the log events in the log event set using different segmentation granularities to obtain preliminary log event chain sets corresponding to the different segmentation granularities;

[0027] Counting the word frequency and inverse file frequency of each preliminary selected log event chain in the log data in the preliminary selected log event chain set corresponding to different segmentation granularities;

[0028] Sort the preliminary selected log event chains of the preliminary selected log event chain sets corresponding to different segmentation granularities according to the word frequency inverse file frequency of each preliminary selected log event chain;

[0029] Matching and pruning are performed on the preliminary selected log event chains in the sorted preliminary selected log event chain sets corresponding to different segmentation granularities, and repeated preliminary selected log event chains are removed to obtain the log event chains corresponding to the log data.

[0030] In some embodiments of the present invention, matching and pruning the pre-selected log event chains in the sorted pre-selected log event chain sets corresponding to different segmentation granularities includes:

[0031] Delete the preliminary log event chains ranked after the specified ranking in the preliminary log event chain sets corresponding to different segmentation granularities to obtain the filtered log event chain sets corresponding to different segmentation granularities;

[0032] Starting from the maximum granularity, the filtered log event chain sets corresponding to each granularity are sequentially used as the target log event chain set, the filtered log event chains in the filtered log event chain set with low granularity are matched with the filtered log event chains in the target log event chain set, and the filtered log event chains that are repeated with the filtered log event chains in the target log event chain set are deleted from the filtered log event chain set with low granularity, until the filtered log event chain set corresponding to the lowest granularity is matched, thereby obtaining the log event chain corresponding to the log data.

[0033] Another aspect of the present invention provides a log data event graph construction device, the device comprising:

[0034] The time extraction module is used to treat the first line of the document that conforms to the time format in the received log data as a log event and extract the time information corresponding to each log event;

[0035] A trigger word extraction module is used to extract the trigger word corresponding to each log event according to the log statement corresponding to each log event in the log data;

[0036] An argument extraction module is used to classify log events in the log data according to the trigger words corresponding to each log event, divide the log events containing the same trigger words into a log event set, and extract the arguments corresponding to each log event based on each log event set;

[0037] An event chain building module, configured to build a log event chain for each log event in the log data according to time information corresponding to each log event in the log data;

[0038] The graph creation module is used to construct an event graph of the log data based on the log event chain and the time information, trigger words, and arguments corresponding to each log event.

[0039] Another aspect of the present invention provides a log data event graph construction device, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the above-mentioned log data event graph construction method.

[0040] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned log data event graph construction method when executed by a processor.

[0041] The present invention provides a log data event graph construction method, device, equipment and storage medium. Based on the time characteristics of log events, log data is parsed to extract the time information, trigger words and arguments of log events. Based on the time characteristics of log events, a log event chain is constructed. Based on the log event chain, the time information, trigger words and arguments of log events, an event graph of log data is created. The graph can represent the logical relationship of log events, facilitate query use, and improve the efficiency and accuracy of data query. At the same time, the graph creation process does not require manual labeling, which improves the efficiency of log data processing.

[0042] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0043] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are intended to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are merely for the purpose of illustrating the principles of the present invention. To facilitate the illustration and description of certain portions of the present invention, corresponding portions in the drawings may be exaggerated, that is, may be larger than other components in an exemplary device actually manufactured according to the present invention. In the drawings:

[0045] Figure 1 This is a flow chart of a method for constructing a log data event graph provided in one embodiment of this specification;

[0046] Figure 2 This is a flow chart of constructing a log event graph in one embodiment of this specification;

[0047] Figure 3 This is a schematic diagram of the module structure of an embodiment of the log data event graph construction device provided in this specification;

[0048] Figure 4 This is a hardware structure block diagram of a log data event graph construction server in one embodiment of this specification. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0050] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0051] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0052] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0053] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0054] With the advancement of technology, the amount of data in log systems is surging. When it is necessary to query log data, how to accurately and quickly query the desired information to improve system maintenance efficiency and thus improve system performance is a problem that needs to be faced.

[0055] The embodiments of this specification provide a method for constructing a log data event graph. The method extracts the time information, trigger words, and arguments of each log event in the log data, and constructs a log event chain for each log event in the log data based on the time information of each log event. The method then creates a log event graph based on the time information, trigger words, arguments, and log event chain of each log event in the log data. The log event graph can clearly represent key information of each log event in the log data, such as time, trigger words, and arguments, and can also represent the logical relationships between each log event, thereby facilitating data query and improving the efficiency and accuracy of log data query.

[0056] Figure 1 This is a flow chart of a method for constructing a log data event graph provided in one embodiment of this specification. Figure 1 As shown, in one embodiment of the method for constructing a log data event graph provided in this specification, the method can be applied to terminal devices such as computers, tablet computers, servers, smart phones, and smart wearable devices. The method may include the following steps:

[0057] Step 102: The first line of the document that conforms to the time format in the received log data is regarded as a log event, and the time information corresponding to each log event is extracted.

[0058] In the specific implementation process, generally, a log data will contain a large amount of data, which may include multiple log events. The embodiment of this specification can parse the received log data based on the time feature and extract the time information corresponding to each log event. Generally, the printing of standard logs starts with the time or timestamp, but some log data is very long and may contain symbols such as line breaks, so each line of the document cannot be regarded as a log print, that is, as a log event. In the embodiment of this specification, the first line of the document that conforms to the time format is treated as a log print as a log event according to the time feature. Through this document parsing method, the received log data is restored to the log print state each time, and the parsed time information is finally used as the occurrence time of each log event.

[0059] Step 104: Extract the trigger word corresponding to each log event based on the log statement corresponding to each log event in the log data.

[0060] In the specific implementation process, the trigger word of the log event can represent the category attribute of the log event. When using the standard log component, each line of the log represents a log event. Each line of the log is generally triggered by a class defined by the system. For example, if there is a class named "web.controller", the event trigger word in the log is this class. The embodiment of this specification extracts the event trigger word in a line of log. Most programming languages such as python, c++, c#, java, etc. will use common log components such as log, log4j, logback, etc. to print log output. The logs printed by these components generally comply with standard log specifications. The log content includes log-time, log-info, log-deug, log-warn, log-erro, log-trace, etc. Table 1 is an example of log data given in an embodiment of this specification. As shown in Table 1, RequestID in Sample 3, spring-cloud-monito in Sample 4, and className in Sample 5 can represent the trigger words in the corresponding samples.

[0061] Table 1

[0062]

[0063] In the embodiments of this specification, an intelligent learning model can be used to learn and identify trigger words in log events. For example, the time data in the log data obtained by parsing in the above embodiments is removed, and a line of log is set to d. D must contain a trigger word and a description of the event. For the same log system, because the log components used are the same and the format within the system is predefined, it is believed that the log format of the current system is fixed, and the event trigger words are also relatively fixed in the format. Machine learning methods and neural networks can be used to learn the log format of the system in an unsupervised manner, and then the trained intelligent learning model can be used to predict the trigger words in the log events.

[0064] In some embodiments of this specification, extracting the trigger word corresponding to each log event includes:

[0065] Segmenting the log statements of each log event in the log data to obtain a segmentation set corresponding to each log event;

[0066] Counting the word frequency inverse file frequency of each word in the word segmentation set of each log event in the log data;

[0067] Sort the words in the word set of each log event according to the word frequency inverse file frequency, cluster the sorted word set of each log event, and obtain a trigger word list corresponding to each log event;

[0068] The log data is annotated with keywords based on the trigger word list, and a trigger word prediction model is trained using the annotated log data. The trigger word prediction model is used to predict the trigger word of each log event in the log data.

[0069] In a specific implementation process, when extracting trigger words from log events, the log statements of each log event in the log data may be first segmented to obtain a segmentation set corresponding to each log event.

[0070] In some embodiments of the present specification, before performing word segmentation on the log statement of each log event in the log data, the method further includes:

[0071] Segment each log event into sentences based on specified characters, and delete characters that have no semantic meaning in each segmented sentence to obtain the filtered log statement corresponding to each log event;

[0072] Perform word segmentation on the filter log statement corresponding to each log event.

[0073] In the specific implementation process, since the log contains a variety of symbols, the sentences are first segmented using separators such as symbols (\n, \t, @, ...). Since the log contains Chinese and English data, and numbers, etc., the embodiment of this specification can first identify the English characters that do not belong to the English word list, such as "bxcdadd" and other strings, as a whole according to the English word list. The same method can be used to identify the mixed English and number, or pure number strings as a whole: such as "4499335552312" or "5g3jwe8y81ssd", and these strings that do not contain semantic information are removed. Among them, not having semantic meaning can be understood as English characters that do not belong to English words, as well as combinations of English and number or pure numbers. Such characters generally do not have specific semantic meanings and will not be trigger words for log events. After deleting these characters that do not have semantic meanings, the remaining strings are segmented using the English word list and Chinese word segmentation tools to obtain the word segmentation set corresponding to each log event. For example, the word segmentation result example is: ["className", "This", "is", "debug", "message", accepted, successful"]. By segmenting the log statement into sentences, characters without semantic meaning are deleted. Characters that do not belong to trigger words are eliminated, which reduces the workload for subsequent trigger word extraction and improves data processing speed.

[0074] After obtaining the word segmentation set corresponding to each log event, a statistical method can be used to extract possible trigger words for the log event. Each log line is a log event, which must contain its event trigger word. This word is the keyword to be extracted in this embodiment of the specification, and these keywords appear multiple times throughout the entire log document. This embodiment of the specification can use TF-IDF and text-rank methods to generate a weighted ranking result for the keywords of the entire log document D: W = {w1, w2, w3, w4...wm}, such as: where w1 = spring: 0.05, w2 = boot: 0.049. TF-IDF (termfrequency-inverse document frequency) is a commonly used weighting technique used in information retrieval and text mining. TF-IDF is a statistical method used to assess the importance of a word to a document set or a document in a corpus. The importance of a word increases proportionally with the number of times it appears in a document, but decreases inversely with the frequency of its appearance in the corpus. The TextRank algorithm is a graph-based ranking algorithm for keyword extraction and document summarization.

[0075] Using the TF-IDF and text-rank methods, we can count the word frequency inverse file frequency, i.e., weight, of each word in the word set corresponding to each log event in the log data. Based on the weight corresponding to each word, we can sort the words in the word set of each log event, and then cluster the sorted word sets using clustering methods such as the Gaussian Mixed Model. Assuming that the clustering category is k, retain the cluster data with the highest cluster center value to obtain the trigger word table W = {w1, w2, w3, w4....wk} corresponding to each log event.

[0076] Then, using the remote supervision method and trigger word table W, each line of log data d is labeled, the keywords in d are labeled, and the model is trained to learn the distribution of trigger words in the log sentence d. The model can select a previously pre-trained language model, use the Bert For Token Classification (label classification method) + CRF (conditional random field algorithm, conditional random field algorithm) method to predict the input log sentence, and predict the trigger words corresponding to each log event. Among them, because the data characteristics of the log are very different from the general language model, the embodiment of this specification can obtain a large number of standard logs from different devices and different development environments to train the language model of the log system before extracting the trigger words and arguments of the log event, use the pre-trained language model to learn the data distribution of a large number of logs, and then use the trained language model to achieve better performance in the next tasks such as: extraction of trigger words and arguments.

[0077] Using keyword extraction and remote supervision models to extract event trigger words eliminates the need for manual templates, rules, or labeled data, speeding up log data processing. Of course, when labeling log event keywords, manual labeling of some event trigger words can also be used to improve trigger word extraction accuracy.

[0078] In some embodiments of this specification, using the trigger word prediction model to predict the trigger word for each log event in the log data includes:

[0079] Using the trigger word prediction model to predict candidate trigger words and scores corresponding to the candidate trigger words for each log event in the log data, and using the candidate trigger word with the highest score as the trigger word for the corresponding log event;

[0080] If the scores of the candidate trigger words of the log event predicted by the trigger word prediction model are all less than the preset scores, the word frequency inverse file frequencies of the candidate trigger words of the log event in the log data are counted, and the candidate trigger words are sorted and clustered based on the word frequency inverse file frequencies to obtain a new trigger word list for each log event. The trigger words of each log event are re-predicted based on the new trigger word list and the trigger word prediction model until the score of the trigger word of each log event is greater than or equal to the preset score.

[0081] In the specific implementation process, the trigger word prediction model can be used to predict the candidate trigger words and their corresponding scores for each log event in the input log data, and the predicted candidate trigger word with the highest score for each sentence is output as the trigger word for the log event corresponding to the sentence. If the log data is poor and the scores of the predicted candidate trigger words are all less than the preset scores, the predicted candidate trigger words can be used as the segmentation set of the log event, and the steps of the above embodiment can be repeated. For example: when the trigger word prediction model is used to predict the trigger words of each log event in the log data, the scores of the candidate trigger words of each log event output by the model are all less than the preset scores, then the output candidate trigger words are used as the new segmentation set of the log event, and the TF-IDF and text-rank methods are used to count the word frequency inverse file frequency, i.e., weight, of the candidate trigger words in the log data, and the candidate trigger words are sorted and clustered based on the statistical results to obtain a new trigger word list for the log event. We use the new trigger word list for log events to annotate the log statements of the log events. We then use the annotated log statements to train a trigger word prediction model. This trained trigger word prediction model predicts trigger words for log events until we extract the event trigger word wd for each log line d. Ultimately, we obtain D = {d1, d2, ..., dn}. If d1 has the event trigger word w1, we describe that log line as an event (Event) E1 = (d1, w1).

[0082] When the prediction results are poor, the prediction results are used to iteratively re-predict the trigger words of the log event, which improves the accuracy of trigger word extraction and lays an accurate data foundation for the creation of subsequent log event maps.

[0083] Step 106: Classify the log events in the log data according to the trigger words corresponding to each log event, divide the log events containing the same trigger words into a log event set, and extract the arguments corresponding to each log event based on each log event set.

[0084] In specific implementations, arguments can be understood as substantive components with argument roles. Argument roles are semantic roles assigned to noun phrases by predicates based on their semantic relationships with related noun phrases. Specifically, arguments used with a verb play a semantic role in the action or state represented by the verb. For example, a health check event with the trigger word "HealthCheck" may have arguments such as "Success," "Error," and "TimeOut." In this embodiment, log events can be classified based on their corresponding trigger words and arguments for each log event can be extracted based on the classification results. Given a class of events Ex, log sentences with the same event trigger words are extracted and considered to be of the same class. Multiple log event sets are obtained, and we can obtain Ex = ({dx1, dx2, dx3...dxn}, wx). The fact that events in a log event set share the same trigger words indicates that the log events in the set are of the same class. Arguments can be extracted by learning the sentences associated with each log event in the set.

[0085] In some embodiments of this specification, extracting arguments corresponding to each log event includes:

[0086] The log statements of the log events in each log event set are input into the clustering model respectively, the log events in each log event set are semantically clustered, and the semantic vector of each cluster center cluster is used as the argument corresponding to each log event in the corresponding log event set.

[0087] In a specific implementation, a set of log events, such as {dx1, dx2, dx3...dxn}, is input into a clustering model for semantic clustering. The resulting classification data is the number of arguments of the log events in the set. The semantic vector of each central cluster is recorded as the argument of the log event of that category. For event Ex, the argument set (Argument) Ax = {Ax1, Ax2, Ax3...Axj} is obtained. The clustering model can choose the unsupervised model SCCL (Supporting Clustering with Contrastive Learning). This method uses data augmentation and contrastive learning to perform unsupervised clustering on sentences. For a class of events Ex, the following can be obtained: Ex = ({dx1, dx2, dx3...dxn}, wx, {Ax1, Ax2, Ax3...Axj}). For a log sentence dx1, dx1 = (Ex, wx, Ax1).

[0088] The embodiments of this specification use a method based on SCCL clustering to cluster event arguments. This method offers higher clustering accuracy and is unsupervised, requiring no manual labeling or rule definition. This improves log data processing speed and reduces log data processing costs. Of course, if needed, some arguments can also be manually labeled to improve clustering accuracy.

[0089] Of course, you can also pre-acquire the print structure of the log as needed, and extract trigger words and arguments based on the log structure.

[0090] Step 108: Construct a log event chain for each log event in the log data according to the time information corresponding to each log event in the log data.

[0091] In the specific implementation process, the time information corresponding to the log event can be understood as the time when the log event occurred. Log information has a strong temporal feature. Log data is generally typed out line by line according to the order of occurrence of events. The time information represents the sequential relationship of the occurrence of each event. The embodiment of this specification can automatically extract the sequential relationship of events based on the time characteristics of the log event and construct a log event chain for each log event in the log data.

[0092] In some embodiments of this specification, constructing a log event chain for each log event in the log data according to time information corresponding to each log event in the log data includes:

[0093] Sort the log events in the log data according to the time information corresponding to each log event in the log data to obtain a log event set;

[0094] Segmenting the log events in the log event set using different segmentation granularities to obtain preliminary log event chain sets corresponding to the different segmentation granularities;

[0095] Counting the word frequency and inverse file frequency of each preliminary selected log event chain in the log data in the preliminary selected log event chain set corresponding to different segmentation granularities;

[0096] Sort the preliminary selected log event chains of the preliminary selected log event chain sets corresponding to different segmentation granularities according to the word frequency inverse file frequency of each preliminary selected log event chain;

[0097] Matching and pruning are performed on the preliminary selected log event chains in the sorted preliminary selected log event chain sets corresponding to different segmentation granularities, and repeated preliminary selected log event chains are removed to obtain the log event chains corresponding to the log data.

[0098] In the specific implementation process, for a log document D = {d1, d2, d3...dn}, it can be expressed as E = {e1, e2, e3....em}, and each event e has the event occurrence time parsed in the first step, denoted as T, then there is (e1, t1). Assume that there is an event chain Chain = (ec1, ec2, ec3, ..., eck) in the document. For example, a database operation event E = "Database" has arguments "Accept DATA", "Write DATA", and "Return Success". The arguments of these three events occur in sequence to form an event chain that accepts data, adds it to the database, and returns successfully. The purpose of the embodiments of this specification is to extract these event chains with sequential logical relationships. For the log document E = {e1, e2, e3....en}, these events can be divided into blocks of different granularities according to different segmentation granularities, namely n-gram. For example, for a 2-gram segmentation method, the event block is C2 = [C2-1, C2-2..., C2-n], where C2-1 = [e1, e2]; for a 5-gram event block, C5 = [C5-1, C5-2..., C5-n], where C5-1 = [e1, e2, e3, e4, e5]. In one embodiment of this specification, a segmentation granularity from 2-gram to 8-gram can be used. Assuming the number of gram is m, each segmented event chain Cm-x = [ex, ex+1, ..., ex+m] is treated as a sentence. The log document D is segmented according to the number of days to obtain i documents D = {D1, D2, D3, ..., Di} and document Di = {ei-1, ei-2, ei-3....ei-n}. The TF and IDF of Cm-x in {D1, D2, D3, ..., Di} are counted and recorded as the statistical score of the chain and sorted. Then, the long-tail data in the sorting, i.e., the data with the lowest score, is removed to obtain the ranking of the event chain under each gram, Cm-1, Cm-2, Cm-3...Cm-k. Finally, according to the longest match principle, each Cm-x is pruned layer by layer from 8-gram to 2-gram to obtain the longest non-repeating event chain: C1, C2, C3...Ck.

[0099] For example, after sorting the events in the log data by time, there is a log event set {ABCDEFG}. The log event set is segmented using different segmentation granularities, such as 2-gram segmentation, to obtain a set of preliminary log event chains {AB, BC, CD, DE, EF, FG} corresponding to the 2-gram segmentation granularity. The same applies to other segmentation granularities until the highest segmentation granularity is reached. The selection of segmentation granularity can be determined according to actual needs and is not specifically limited in the embodiments of this specification. Then, each preliminary log event chain in the preliminary log event chain set is searched in the entire document, and the word frequency and reverse file frequency corresponding to each preliminary log event chain in different segmentation granularities are counted. Based on the statistical results, the preliminary log event chains in each preliminary log event chain set are sorted. Then, the preliminary log event chains in each sorted preliminary log event chain set are matched and pruned to remove duplicate preliminary log event chains. For example, if ABC contains AB, AB is removed. By analogy, the log event chain corresponding to the log data is obtained.

[0100] In some embodiments of this specification, matching and pruning the pre-selected log event chains in the sorted pre-selected log event chain sets corresponding to different segmentation granularities includes:

[0101] Delete the preliminary log event chains ranked after the specified ranking in the preliminary log event chain sets corresponding to different segmentation granularities to obtain the filtered log event chain sets corresponding to different segmentation granularities;

[0102] Starting from the maximum granularity, the filtered log event chain sets corresponding to each granularity are sequentially used as the target log event chain set, the filtered log event chains in the filtered log event chain set with low granularity are matched with the filtered log event chains in the target log event chain set, and the filtered log event chains that are repeated with the filtered log event chains in the target log event chain set are deleted from the filtered log event chain set with low granularity, until the filtered log event chain set corresponding to the lowest granularity is matched, thereby obtaining the log event chain corresponding to the log data.

[0103] In a specific implementation process, when statistically sorting the preliminary selected log event chains in the preliminary selected log event chain set of each segmentation granularity, the preliminary selected log event chains with low scores and low rankings can be deleted based on the scores of each preliminary selected log event chain to obtain the screening log event chain set of each segmentation granularity. Then, starting from the maximum segmentation granularity, the screening log event chain set corresponding to each segmentation granularity is sequentially used as the target log event chain set. For example, if in one embodiment, the segmentation granularity is 2-6, the 6-gram screening log event chain set can be used as the target log event chain set first, and the screening log event chain of the 2-5 segmentation granularity can be matched with the screening log event chain in the 6-gram screening log event chain set. If there is a duplicate in the 6-gram screening log event chain, it will be deleted, such as: ABCDEF contains ABCD, then ABCD will be deleted. And so on, until the 2-gram screening log event chain set is used as the target log event chain set, the log event chain of the log data is obtained. It can be seen from the above embodiments that there may be more than one log event chain in a log data, depending on the specific situation, and the embodiments of this specification do not make specific limitations.

[0104] The embodiments of this specification extract and sort the time series information of the logs to obtain the sequential relationship of the occurrence of log events and model the log event chain without the need for manual labeling and prior data. The logical relationship of log events in the log data can be obtained, thereby improving the accuracy of the construction of the log data event graph. The log event graph is created based on the log event chain, and the graph can more clearly express the internal logic of the log data, making it easier to query and use.

[0105] Step 110: Construct an event graph of the log data based on the log event chain and the time information, trigger words, and arguments corresponding to each log event.

[0106] In the specific implementation process, after obtaining the log event chain in the log data, an event graph of the log data can be constructed based on the log event chain and the time information, trigger words, and arguments corresponding to each log event. Among them, time, keywords, and arguments are internal elements of each event, and the event chain describes the relationship between events. The event graph of log data can include multiple nodes, each of which can be an event. The attribute value of each node can include the time, trigger words, and arguments of the event. Based on the event chain, the relationship between each node is established, and then the event graph of the log data is constructed.

[0107] The log data event graph construction method provided in the embodiments of this specification parses log data based on the time characteristics of log events, extracts the time information, trigger words, and arguments of log events, and constructs a log event chain based on the time characteristics of log events. Based on the log event chain, the time information, trigger words, and arguments of log events, an event graph of log data is created. This graph can represent the logical relationship between log events, facilitates query use, and improves the efficiency and accuracy of data query. At the same time, the graph creation process does not require manual labeling, which improves the efficiency of log data processing.

[0108] Figure 2 This is a flow chart of the construction of a log event graph in one embodiment of this specification. Figure 2 As shown in the figure, the whole process can be summarized as follows: receiving documents output by standard logs; parsing the received documents according to time features; collecting log data to pre-train the language model; extracting event trigger words; extracting event arguments; extracting event chains; and automatically building a log event graph. Figure 2 The specific process of each step is described in the above embodiment and will not be repeated here.

[0109] The embodiments of this specification mainly focus on the event information of the log, extract the event trigger words, time series information, and event arguments in the log, calculate the sequential relationship between events and construct an event chain. Previous tasks modeled log data as knowledge entities, while the embodiments of this specification model logs as events, not only modeling single events, but also modeling event chains, modeling the relationships and links between events, and then establishing an event graph that can characterize the association relationship between events, facilitating subsequent data queries. In addition, the embodiments of this specification use an unsupervised method to automatically extract the information therein without the need for manual labeling or rule definition, thereby improving the efficiency and accuracy of data processing. The time in the standard log output format is the time when the event occurred, and the log class object type or interface type of the log output is the keyword or trigger word of the event. The embodiments of this specification are oriented to unknown devices and systems, accept the standard log output format, and use an unsupervised method for extraction, without the need for manual labeling or definition of template rules.

[0110] In this specification, the various embodiments of the above method are described in a progressive manner. The same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. For relevant parts, refer to the partial description of the method embodiment.

[0111] Based on the log data event graph construction method described above, one or more embodiments of this specification also provide a device for constructing a log data event graph. The device may include a device (including a distributed system), software (application), modules, components, servers, clients, etc. that use the method described in the embodiments of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of this specification is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the device are similar, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware for predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0112] Specifically, Figure 3 This is a schematic diagram of the module structure of an embodiment of the log data event graph construction device provided in this specification. Figure 3 As shown, the device provided in this specification may include:

[0113] The time extraction module 31 is used to treat the first line of the document that conforms to the time format in the received log data as a log event, and extract the time information corresponding to each log event;

[0114] A trigger word extraction module 32 is used to extract the trigger word corresponding to each log event according to the log statement corresponding to each log event in the log data;

[0115] an argument extraction module 33 for classifying log events in the log data according to trigger words corresponding to each log event, dividing log events containing the same trigger word into a log event set, and extracting arguments corresponding to each log event based on each log event set;

[0116] An event chain building module 34 is used to build a log event chain of each log event in the log data according to the time information corresponding to each log event in the log data;

[0117] The graph creation module 35 is used to construct an event graph of the log data according to the log event chain and the time information, trigger words, and arguments corresponding to each log event.

[0118] In some embodiments of this specification, a log data event graph construction device is further provided, including a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the log data event graph construction method described in the above embodiments, such as:

[0119] The first line of the received log data that conforms to the time format is treated as a log event, and the time information corresponding to each log event is extracted;

[0120] Extracting trigger words corresponding to each log event based on the log statements corresponding to each log event in the log data;

[0121] Classifying the log events in the log data according to the trigger words corresponding to the respective log events, dividing the log events containing the same trigger words into a log event set, and extracting arguments corresponding to the respective log events based on the respective log event sets;

[0122] Constructing a log event chain for each log event in the log data according to the time information corresponding to each log event in the log data;

[0123] An event graph of the log data is constructed based on the log event chain and the time information, trigger words, and arguments corresponding to each log event.

[0124] It should be noted that the above-mentioned apparatus and device may also include other implementations according to the description of the method embodiment. Specific implementations can refer to the description of the relevant method embodiment and will not be described in detail here.

[0125] The method embodiments provided in the embodiments of this specification can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 4 This is a hardware structure block diagram of a log data event graph construction server in one embodiment of this specification. The computer terminal can be the log data event graph construction server or the log data event graph construction device in the above embodiment. Figure 4 The server 10 shown may include one or more (only one is shown in the figure) processors 100 (the processor 100 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a non-volatile memory 200 for storing data, and a transmission module 300 for communication functions. It will be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4More or fewer components shown in the figure may also include other processing hardware, such as a database or multi-level cache, GPU, or other hardware with Figure 4 Different configurations shown.

[0126] The non-volatile memory 200 can be used to store software programs and modules of application software, such as the program instructions / modules corresponding to the log data event map construction method in the embodiment of this specification. The processor 100 executes various functional applications and resource data updates by running the software programs and modules stored in the non-volatile memory 200. The non-volatile memory 200 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the non-volatile memory 200 may further include a memory remotely located relative to the processor 100, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The transmission module 300 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of a computer terminal. In one embodiment, the transmission module 300 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 300 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0128] Corresponding to the above method, the present invention also provides an apparatus, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the apparatus implements the steps of the method described above.

[0129] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0130] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0131] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0132] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0133] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for constructing a log data event graph, characterized in that: The method comprises: The first line of the received log data that conforms to the time format is treated as a log event, and the time information corresponding to each log event is extracted; Extracting trigger words corresponding to each log event based on the log statements corresponding to each log event in the log data; Classifying the log events in the log data according to the trigger words corresponding to the respective log events, dividing the log events containing the same trigger words into a log event set, and extracting arguments corresponding to the respective log events based on the respective log event sets; Constructing a log event chain for each log event in the log data according to the time information corresponding to each log event in the log data; Constructing an event graph of the log data based on the log event chain and the time information, trigger words, and arguments corresponding to each log event; The step of constructing a log event chain for each log event in the log data according to time information corresponding to each log event in the log data includes: Sort the log events in the log data according to the time information corresponding to each log event in the log data to obtain a log event set; Segmenting the log events in the log event set using different segmentation granularities to obtain preliminary log event chain sets corresponding to the different segmentation granularities; Counting the word frequency and inverse file frequency of each preliminary selected log event chain in the log data in the preliminary selected log event chain set corresponding to different segmentation granularities; Sort the preliminary selected log event chains of the preliminary selected log event chain sets corresponding to different segmentation granularities according to the word frequency inverse file frequency of each preliminary selected log event chain; Matching and pruning are performed on the preliminary selected log event chains in the sorted preliminary selected log event chain sets corresponding to different segmentation granularities, and repeated preliminary selected log event chains are removed to obtain the log event chains corresponding to the log data.

2. The method according to claim 1, characterized in that Extracting the trigger words corresponding to each log event includes: Segmenting the log statements of each log event in the log data to obtain a segmentation set corresponding to each log event; Counting the word frequency inverse file frequency of each word in the word segmentation set of each log event in the log data; Sort the words in the word set of each log event according to the word frequency inverse file frequency, cluster the sorted word set of each log event, and obtain a trigger word list corresponding to each log event; The log data is annotated with keywords based on the trigger word list, and a trigger word prediction model is trained using the annotated log data. The trigger word prediction model is used to predict the trigger word of each log event in the log data.

3. The method according to claim 2, characterized in that Before segmenting the log statements of each log event in the log data, the method further includes: Segment each log event into sentences based on specified characters, and delete characters that have no semantic meaning in each segmented sentence to obtain the filtered log statement corresponding to each log event; Perform word segmentation on the filter log statement corresponding to each log event.

4. The method according to claim 2, characterized in that The method of using the trigger word prediction model to predict the trigger word of each log event in the log data includes: Using the trigger word prediction model to predict candidate trigger words and scores corresponding to the candidate trigger words for each log event in the log data, and using the candidate trigger word with the highest score as the trigger word for the corresponding log event; If the scores of the candidate trigger words of the log event predicted by the trigger word prediction model are all less than the preset scores, the word frequency inverse file frequencies of the candidate trigger words of the log event in the log data are counted, and the candidate trigger words are sorted and clustered based on the word frequency inverse file frequencies to obtain a new trigger word list for each log event. The trigger words of each log event are re-predicted based on the new trigger word list and the trigger word prediction model until the score of the trigger word of each log event is greater than or equal to the preset score.

5. The method according to claim 1, wherein The extraction of arguments corresponding to each log event includes: The log statements of the log events in each log event set are input into the clustering model respectively, the log events in each log event set are semantically clustered, and the semantic vector of each cluster center cluster is used as the argument corresponding to each log event in the corresponding log event set.

6. The method according to claim 1, characterized in that The matching and pruning of the pre-selected log event chains in the sorted pre-selected log event chain sets corresponding to different segmentation granularities includes: Delete the preliminary log event chains ranked after the specified ranking in the preliminary log event chain sets corresponding to different segmentation granularities to obtain the filtered log event chain sets corresponding to different segmentation granularities; Starting from the maximum granularity, the filtered log event chain sets corresponding to each granularity are sequentially used as the target log event chain set, the filtered log event chains in the filtered log event chain set with low granularity are matched with the filtered log event chains in the target log event chain set, and the filtered log event chains that are repeated with the filtered log event chains in the target log event chain set are deleted from the filtered log event chain set with low granularity, until the filtered log event chain set corresponding to the lowest granularity is matched, thereby obtaining the log event chain corresponding to the log data.

7. A device using the log data event graph construction method according to any one of claims 1 to 6, characterized in that: The device comprises: The time extraction module is used to treat the first line of the document that conforms to the time format in the received log data as a log event and extract the time information corresponding to each log event; A trigger word extraction module is used to extract the trigger word corresponding to each log event according to the log statement corresponding to each log event in the log data; An argument extraction module is used to classify log events in the log data according to the trigger words corresponding to each log event, divide the log events containing the same trigger words into a log event set, and extract the arguments corresponding to each log event based on each log event set; An event chain building module, configured to build a log event chain for each log event in the log data according to time information corresponding to each log event in the log data; The graph creation module is used to construct an event graph of the log data based on the log event chain and the time information, trigger words, and arguments corresponding to each log event.

8. A log data event graph construction device, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images