Method and apparatus for training log anomaly detection model

Abstract

The application provides a log anomaly detection model training method and device, the log anomaly detection model training method comprises the following steps: obtaining multiple sample log data generated by a signal system; analyzing the multiple sample log data based on an event template to obtain a first template sequence; sorting multiple log templates based on an edit distance between the multiple log templates and a word vector similarity between multiple parsed sample log data to obtain a second template sequence, adding statistical information of a first keyword and a target log template to the second template sequence to obtain a first count matrix; inputting the first count matrix into a clustering model for hierarchical clustering training, and obtaining a log anomaly detection model after the training is completed. The method can increase the dimension of the sample log data corresponding template sequence, thereby improving the training efficiency of the clustering model and the classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of data anomaly detection technology, and in particular to a training method and apparatus for a log anomaly detection model. Background Technology

[0002] As the scale and complexity of signaling systems in urban rail transit continue to increase, the amount of logs generated per unit time is increasing dramatically. Timely monitoring and early warning of log data generated under abnormal operating scenarios can prevent signaling system collapse and reduce economic losses.

[0003] In related technologies, relying solely on manual analysis of signal systems based on professional experience is inefficient and results in high maintenance costs. When using detection models to detect anomalies in various log formats generated by signal systems, the log data is unstructured, with inconsistent time formats and different manufacturers' custom terms or abbreviations, which increases the difficulty of log analysis. Furthermore, the detection models have poor feature representation capabilities, leading to low accuracy in detecting abnormal logs. Summary of the Invention

[0004] This invention provides a training method and apparatus for a log anomaly detection model, which addresses the shortcomings of existing technologies in handling massive log messages, such as low efficiency in judging system faults by human experience and low accuracy in detecting log anomalies by detection models. This invention improves the accuracy and efficiency of detecting abnormal logs.

[0005] This invention provides a training method for a log anomaly detection model, comprising:

[0006] Acquire various sample log data generated by the signal system;

[0007] The various sample log data are parsed based on event templates to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed sample log data corresponding to the target parameter. The event template is determined based on unstructured log data. The target parameter is used to represent the category information of the sample log data.

[0008] The log templates are sorted based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence. A first keyword and statistical information of the target log template are added to the second template sequence to obtain a first counting matrix. The target log template is the log template that stores the most parsed sample log data among the multiple log templates. The first keyword is obtained based on the printing parameters in the multiple sample log data.

[0009] The first counting matrix is ​​input into the clustering model for hierarchical clustering training, and the log anomaly detection model is obtained after training.

[0010] According to a training method for a log anomaly detection model provided by the present invention, the step of inputting the first counting matrix into a clustering model for hierarchical clustering training, and obtaining the log anomaly detection model after training, includes:

[0011] The first counting matrix is ​​input into a clustering model for clustering, resulting in multiple sample clusters;

[0012] The target distance between any two sample clusters in the plurality of sample clusters is obtained based on the mean clustering formula. When the target distance is less than a first threshold, the two sample clusters are merged to obtain a fused cluster.

[0013] The log anomaly detection model is obtained when the target distance is greater than the first threshold or the number of fusion clusters is less than the second threshold.

[0014] According to a training method for a log anomaly detection model provided by the present invention, the step of inputting the first counting matrix into a clustering model for hierarchical clustering training, and obtaining the log anomaly detection model after training, further includes:

[0015] The counting matrix is ​​normalized and vectorized to obtain the counting vector;

[0016] The counting vector is input into the clustering model for hierarchical clustering training, and the log anomaly detection model is obtained after training.

[0017] According to the training method of the log anomaly detection model provided by the present invention, the step of determining the event template includes:

[0018] Obtain the token object from the unstructured log data;

[0019] The event template is obtained by classifying the unstructured log data based on the token object and the second keyword in the unstructured log data.

[0020] According to the training method of the log anomaly detection model provided by the present invention, the mean clustering formula includes:

[0021]

[0022] Among them, C i and C j They are any two sample clusters from the plurality of sample clusters. C i The average value, Cj The average value.

[0023] The present invention also provides a log anomaly detection method, comprising:

[0024] Obtain the log data to be tested;

[0025] The log data to be detected is parsed based on the event template to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed log data to be detected corresponding to the target parameter. The event template is determined based on unstructured log data. The target parameter is used to represent the category information of the log data to be detected.

[0026] The log templates are sorted based on the edit distance between them and the word vector similarity between them. A second template sequence is obtained from the sorted log templates. A first keyword and statistical information of the target log template are added to the second template sequence to obtain a second counting matrix. The target log template is the log template that stores the most parsed log data among the log templates. The first keyword is obtained based on the printing parameters in various sample log data.

[0027] Based on the log anomaly detection model, the second counting matrix is ​​clustered to obtain the target clustering distance. If the target clustering distance is less than a third threshold, the log data to be detected is determined to be abnormal log data.

[0028] This invention provides a training apparatus for a log anomaly detection model, comprising:

[0029] The acquisition module is used to acquire various sample log data generated by the signal system;

[0030] The parsing module is used to parse the various sample log data based on event templates to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed sample log data corresponding to the target parameter. The event template is determined based on unstructured log data. The target parameter is used to represent the category information of the sample log data.

[0031] The feature extraction module is used to sort the multiple log templates based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence, and add a first keyword and statistical information of the target log template to the second template sequence to obtain a first counting matrix. The target log template is the log template that stores the most parsed sample log data among the multiple log templates. The first keyword is obtained based on the printing parameters in the multiple sample log data.

[0032] The clustering module is used to input the first counting matrix into the clustering model for hierarchical clustering training, and to obtain the log anomaly detection model after training.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a training method for a log anomaly detection model as described in any of the preceding claims or the log anomaly detection method.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the log anomaly detection model as described in any of the preceding claims or the log anomaly detection method.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the training method of the log anomaly detection model as described in any of the preceding claims or the log anomaly detection method.

[0036] The training method and apparatus for the log anomaly detection model provided by the present invention obtains a first template sequence by parsing sample log data according to an event template, adds template similarity information and statistical analysis information to the first template sequence, and performs statistical classification using a first keyword, thereby increasing the dimensionality of the first template sequence and improving the training efficiency and classification accuracy of the clustering model. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the training method of the log anomaly detection model provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the interface for parsing sample log data based on event templates provided by the present invention;

[0040] Figure 3 This is a flowchart illustrating the log anomaly detection method provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the training device for the log anomaly detection model provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] Figure 1 This is a flowchart illustrating the training method of the log anomaly detection model provided by the present invention, as shown below. Figure 1 As shown, the training method of the log anomaly detection model includes steps 110, 120, 130 and 140.

[0045] Step 110: Obtain various sample log data generated by the signal system.

[0046] In this step, the various sample log data can be log data generated by different modules in the communication platform within the signal system during normal operation, as well as logs containing fault information.

[0047] In some embodiments, the collected sample log data can be collected and summarized in the form of text files to serve as input data when training the clustering model in subsequent processes.

[0048] In this embodiment, logs in various formats can be collected and recorded in a file or template in a unified format. After preprocessing steps such as sorting and similarity analysis, the files or templates can be used to train a clustering model.

[0049] In some embodiments, version information of different modules and the correspondence between the log data generated by the modules can be added to the text file to facilitate the subsequent generation of anomaly analysis reports.

[0050] In this embodiment, version information of different modules can be added to the text file. The version information can be represented by a complete version number or a specified identifier (number, serial number or other characters, etc.), which can help determine the log anomalies of other modules.

[0051] Step 120: Parse multiple sample log data based on event templates to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed sample log data corresponding to the target parameter. The event template is determined based on unstructured log data, and the target parameter is used to represent the category information of the sample log data.

[0052] It should be noted that the system kernel, various application servers, and most applications in the signal system may output logs. The content, size, and purpose of these logs vary. These log data often contain a lot of useful information for users, and users can analyze different types of log data to achieve different functional requirements.

[0053] In this step, the target parameter can be the category information corresponding to the sample log data. For example, Receive fromnode blk_0001 and Send 120bytes to blk_0002 belong to two different categories of log data. The target parameter can be keywords with high word frequency in the log data, such as Receive, from, Send, and bytes to.

[0054] In this embodiment, the sample log data is classified into the corresponding event templates for parsing according to the target parameters corresponding to each sample log data, and the corresponding log templates are obtained.

[0055] In this step, the text entries of the log data are usually composed of a preset fixed template part and a specific parameter part. After obtaining a large amount of log data, the event template can be obtained from the unstructured log data through preprocessing operations such as word segmentation technology, and then anomaly detection can be performed on the template sequence.

[0056] In this step, the first template sequence may include multiple log templates, each of which can be used to represent log data in different formats. One log template can represent one or more log data entries. In this embodiment, for log data generated by the signal system, Receive from node blk_0001, the fixed template parts are "Receive", "from", and "node". These fixed template parts can be common or most frequently occurring word segments, while the specific parameter parts such as blk_0001 are usually pure numbers or combinations of numbers and letters or other symbols.

[0057] Step 130: Sort the multiple log templates based on the edit distance between multiple log templates and the word vector similarity between multiple parsed sample log data to obtain a second template sequence, and add the first keyword and the statistical information of the target log template to the second template sequence to obtain a first counting matrix; the target log template is the log template that stores the most parsed sample log data among the multiple log templates, and the first keyword is obtained based on the printing parameters in multiple sample log data.

[0058] In this step, a log template can be randomly selected first, and the edit distance between the template and other templates in the first template sequence and the similarity of word vectors in each event template can be calculated and sorted. Generally speaking, log templates with similar sequence numbers have higher similarity.

[0059] In this step, the printed parameters can be parameters such as Init, Debug, or Warning from the sample log data.

[0060] In this embodiment, it is assumed that the first template sequence contains 10 log templates, and the initial number of each event template is 1 to 10. The templates are renumbered by calculating the edit distance between the templates so that the sequence numbers of two templates with high similarity are close. Before this, in the log collection stage, a log sequence with a fixed window size of 100 is taken to obtain a counting matrix of size 100*10. Mathematical statistics information column and first keyword information column are added to the counting matrix. The mathematical statistics information column can be the sequence number of the log template with the most parsed log data or the most target parameters stored in the first template sequence, and the most frequently occurring log tag class column. For example, the most frequently occurring log template sequence number and the most frequently occurring log tag class column are Init(0), Debug(1) and Warning(2).

[0061] In this embodiment, the first keyword information column can be obtained by calculating the TFIDF-IDF (Term Frequency - Inverse Document Frequency) value of the first template sequence to obtain the three words with the highest TFIDF values, such as "Extension A", "Send", and "Central Server". Then, the Chinese words are converted into numbers using the correspondence between words and sequence numbers in the corpus. The most frequently occurring log template sequence number, the most frequently occurring log tag category column, and three keyword columns are added to the above counting matrix, finally resulting in an information matrix of size 100*15, which is the first counting matrix.

[0062] Step 140: Input the first counting matrix into the clustering model for hierarchical clustering training, and obtain the log anomaly detection model after training.

[0063] In this step, after obtaining the first counting matrix with expanded dimensions from the original counting matrix, the first counting matrix is ​​used as the input of the clustering model for hierarchical clustering training.

[0064] It should be noted that hierarchical clustering creates a multi-level clustering tree by calculating the similarity between data points of different categories; hierarchical decomposition includes two methods: bottom-up decomposition and top-down decomposition. The bottom-up agglomerative method treats all log vector samples as a cluster, and merges the two clusters with the smallest distance in turn, repeating until the termination condition is met; conversely, the top-down splitting method first treats all log vectors as a whole, finds the two clusters with the largest distance in the cluster and splits them, repeating until the number of clusters meets the termination condition.

[0065] In this embodiment, the termination condition may be that the distance between two clusters exceeds a first threshold or the number of clusters exceeds a second threshold.

[0066] In this embodiment, both the first threshold and the second threshold can be customized according to actual needs.

[0067] In some embodiments, other clustering models or algorithms can also be clustered based on the first counting matrix to obtain a usable log anomaly detection model.

[0068] In this embodiment, other clustering models or algorithms may be K-means clustering algorithm, Gaussian mixture clustering model or density clustering algorithm, etc.

[0069] The training method for the log anomaly detection model provided by this invention parses sample log data according to event templates to obtain a first template sequence, adds template similarity information and statistical analysis information to the first template sequence, and performs statistical classification using the first keyword, thereby increasing the dimensionality of the first template sequence and improving the training efficiency and classification accuracy of the clustering model.

[0070] In some embodiments, the first counting matrix is ​​input into a clustering model for hierarchical clustering training, and a log anomaly detection model is obtained after training. This includes: inputting the first counting matrix into a clustering model for clustering to obtain multiple sample clusters; determining the target distance between any two sample clusters in the multiple sample clusters based on the mean clustering formula; when the target distance is less than a first threshold, merging any two sample clusters to obtain a fused cluster; and obtaining the log anomaly detection model when the target distance is greater than the first threshold or the number of fused clusters is less than a second threshold.

[0071] It should be noted that the hierarchical clustering method of the clustering model can be either bottom-up agglomerative hierarchical clustering or top-down splitting hierarchical clustering.

[0072] In this embodiment, the target distance can refer to the average distance between any two sample clusters in a plurality of sample clusters, or it can be the minimum distance or the maximum distance; the corresponding mean clustering formula is used to perform clustering based on the average distance between two sample clusters, or to perform clustering based on the minimum distance or the maximum distance between two sample clusters.

[0073] The following explanation uses a clustering model to illustrate agglomerative hierarchical clustering.

[0074] In this embodiment, the first counting matrix is ​​used as the input for agglomerative hierarchical clustering. Let the number of sample log data contained in the first counting matrix be D = x1, x2, ..., x. m Given a similarity metric function s, the number of clusters k, and clustering samples, perform clustering according to the following steps:

[0075] (1) Initialize each sample log data as a sample cluster, c i =x i i = 1, 2, ..., m;

[0076] (2) Calculate the pairwise distance d(i,j) between the sample log data, where i = 1, 2, ..., m, j = 1, 2, ..., m;

[0077] (3) Using the similarity measurement function s, find the two most similar sample clusters and merge them;

[0078] (4) Stop clustering when the number of fused clusters is less than or equal to k (corresponding to the second threshold), or the target distance between any two sample clusters exceeds the first threshold, and use the clustering model as the available log anomaly detection model.

[0079] In some embodiments, after the clustering model is trained, the resulting log anomaly detection model may contain only two fused sample clusters: a normal sample cluster and an anomaly sample cluster. During training, the number of fused clusters decreases. The log anomaly detection model training method provided by this invention uses a hierarchical clustering training method by inputting the expanded-dimensional technical matrix into the clustering model. The automatic clustering process based on mean distance has low computational cost and can overcome the novel problem of outlier sensitivity.

[0080] In some embodiments, the first counting matrix is ​​input into a clustering model for hierarchical clustering training, and a log anomaly detection model is obtained after training. The method further includes: normalizing and vectorizing the counting matrix to obtain a counting vector; and inputting the counting vector into a clustering model for hierarchical clustering training to obtain a log anomaly detection model.

[0081] In this embodiment, normalizing the first counting matrix can reduce the impact of the event modules with the largest number in the first counting matrix on other event modules with a smaller number.

[0082] In this embodiment, the first counting matrix is ​​vectorized, which can be done by converting the first counting matrix into a vector form so that subsequent processes can measure the distance between the cluster centers of the log vectors corresponding to each sample log data in the first counting vector and the cluster centers of other log vectors.

[0083] The training method for the log anomaly detection model provided by this invention obtains a counting vector that can be used for distance measurement by normalizing and vectorizing the first counting matrix, thereby reducing the interference of data noise in the counting matrix and improving the accuracy of the clustering process.

[0084] In some embodiments, the step of determining the event template includes: obtaining a token object from unstructured log data; classifying the unstructured log data based on the token object and the second keyword corresponding to the unstructured log data to obtain the event template.

[0085] In this embodiment, the number of tokens in the unstructured log data parsing time log data is regarded as an important criterion for extracting the counting template. First, the variable token is converted into specific symbols such as * or other identifiers. Then, the different formats of each sample log data are distinguished by the number of tokens. That is, the sample log data is classified by the number of category tokens. Finally, an event template is obtained based on the second keyword of the sample log.

[0086] Figure 2 This is a schematic diagram of the interface for parsing sample log data based on event templates provided by the present invention. Figure 2 In the illustrated embodiment, for the two log data generated by the signal system: Receive from node blk_0001 and Send 120 bytes to blk_0002, the fixed template part of the first log data is "Receive", "from", and "node". This fixed template part can be common or the most frequently occurring word segmentation, while the specific parameter part blk_0001 is usually a combination of pure numbers or numbers and letters and other symbols; the fixed template part of the second log data is "Send" and "bytes to", and the corresponding specific parameter parts are "120" and "blk_0002". At this time, the event templates corresponding to the two logs can be represented as Receive from node* and Send*bytes to*, respectively.

[0087] The training method for the log anomaly detection model provided by this invention obtains a first template sequence by structuring multiple sample log data through event templates. This sequence can be directly used in the hierarchical clustering process of the clustering model, unifying the format of sample log data and improving log parsing efficiency.

[0088] In some embodiments, the mean clustering formula includes:

[0089]

[0090] Among them, C i and C j These are any two sample clusters from multiple sample clusters. C i The average value, C j The average value, where:

[0091]

[0092] Among them, |C i |For sample cluster C i The number of samples, |C j |For sample cluster C j The number of samples, p is C i For any sample log data in C, q is C j Any sample log data in the database.

[0093] In this embodiment, the target distance can be the average distance between any two sample clusters among multiple sample clusters. The average distance is compared with a first threshold for comparative analysis. When the average distance is less than the first threshold, the two sample clusters are merged to obtain a fused cluster.

[0094] In this embodiment, after sequentially traversing all sample clusters in the first counting matrix, it is determined whether the number of fused clusters exceeds the second threshold or whether the average distance between each fused cluster exceeds the first threshold. If so, the clustering process is stopped, and the log anomaly detection model is obtained. If the above threshold judgment conditions are not met, the clustering model continues to perform hierarchical clustering training until either of the above threshold judgment conditions is met.

[0095] The training method for the log anomaly detection model provided by this invention reduces the computational load of the clustering process and improves clustering efficiency by setting a mean clustering formula to perform hierarchical clustering training on the clustering model.

[0096] Figure 3 This is a flowchart illustrating the log anomaly detection method provided by the present invention, as follows: Figure 3 As shown, the log anomaly detection method includes steps 310, 320, 330 and 340.

[0097] Step 310: Obtain the log data to be tested.

[0098] In this step, the log data to be detected is log data generated by different modules in the communication platform within the signal system during runtime.

[0099] In some embodiments, the collected log data to be detected can be collected in the form of a text file and used as input data when testing the log anomaly detection model in subsequent processes.

[0100] In this embodiment, the log data to be detected can be recorded in a file or template in the same format as the sample log data, and the file or template can be preprocessed such as sorting and similarity analysis to obtain the input data for the log anomaly detection model.

[0101] In some embodiments, version information of different modules and the correspondence between the log data generated by the modules can be added to the text file to facilitate the subsequent generation of anomaly analysis reports.

[0102] In this embodiment, version information of different modules can be added to the text file. The version information can be represented by a complete version number or a specified identifier (number, serial number or other characters, etc.), which can help determine the log anomalies of other modules.

[0103] Step 320: Parse the log data to be detected based on the event template to obtain the first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed log data to be detected corresponding to the target parameter. The event template is determined based on the unstructured log data, and the target parameter is used to represent the category information of the log data to be detected.

[0104] It should be noted that the system kernel, various application servers, and most applications in the signal system may output logs. The content, size, and purpose of these logs vary. These log data often contain a lot of useful information for users, and users can analyze different types of log data to achieve different functional requirements.

[0105] In this step, the target parameter can be the category information corresponding to the log data to be detected. For example, Receivefrom node blk_0001 and Send 120bytes to blk_0002 belong to two different categories of log data. The target parameter can be keywords with high word frequency in the log data, such as Receive, from, Send and bytesto.

[0106] In this embodiment, the log data to be detected is classified into the corresponding event templates for parsing according to the target parameters to obtain the corresponding log templates.

[0107] In this step, the text entries of the log data are usually composed of a preset fixed template part and a specific parameter part. After obtaining a large amount of log data, the event template can be obtained from the unstructured log data through preprocessing operations such as word segmentation technology, and then anomaly detection can be performed on the template sequence.

[0108] In this step, the first template sequence may include multiple log templates, each of which can be used to represent log data in different formats. One log template can represent one or more log data entries. In this embodiment, for log data generated by the signal system, Receive from node blk_0001, the fixed template parts are "Receive", "from", and "node". These fixed template parts can be common or most frequently occurring word segments, while the specific parameter parts such as blk_0001 are usually pure numbers or combinations of numbers and letters or other symbols.

[0109] Step 330: Sort multiple log templates based on the edit distance between multiple log templates and the word vector similarity between multiple parsed log data to be detected, and number the sorted log templates to obtain a second template sequence. Add the first keyword and the statistical information of the target log template to the second template sequence to obtain a second counting matrix. The target log template is the log template that stores the most parsed log data to be detected among the multiple log templates. The first keyword is obtained based on the printing parameters in multiple sample log data.

[0110] In this step, a log template can be randomly selected first, and the edit distance between the template and other templates in the first template sequence and the similarity of word vectors in each event template can be calculated and sorted. Generally speaking, log templates with similar sequence numbers have higher similarity.

[0111] In this step, the printed parameters can be parameters such as Init, Debug, or Warning from the sample log data.

[0112] In this embodiment, when the data to be detected is a log, the corresponding first template sequence includes a log template; when the data to be detected is multiple logs, the corresponding first template sequence includes multiple log templates. A mathematical statistics information column and a first keyword information column are added to the first template sequence corresponding to the data to be detected. The mathematical statistics information column can be the sequence number of the log template with the most parsed log data or the most target parameters stored in the first template sequence, or the most frequently occurring log tag class column. For example, the most frequently occurring log template sequence number and the most frequently occurring log tag class column are Init(0), Debug(1), and Warning(2).

[0113] In this embodiment, the first keyword information column can be obtained by calculating the TFIDF-IDF (Term Frequency - Inverse Document Frequency) value of the first template sequence to obtain the three words with the highest TFIDF values, such as "Extension A", "send", and "central server". Then, the Chinese words are converted into numbers using the correspondence between words and sequence numbers in the corpus. The most frequently occurring log template sequence number, the most frequently occurring log tag category column, and three keyword columns are added to the above counting matrix, and finally, the dimension-expanded information matrix, namely the first counting matrix, is obtained.

[0114] Step 340: Cluster the second counting matrix based on the log anomaly detection model to obtain the target clustering distance, and determine the log data to be detected as abnormal log data if the target clustering distance is less than the third threshold.

[0115] In this step, the method for obtaining the log anomaly detection model is the same as that for obtaining the log anomaly detection model in step 140, and will not be repeated in this embodiment.

[0116] In this step, the target clustering distance is the distance from the centroid of the cluster to the target clustering centroid obtained after inputting the second counting matrix into the log anomaly detection model.

[0117] In this step, the third threshold can be customized according to actual needs.

[0118] In this embodiment, the second counting matrix is ​​input into the log anomaly detection model. When the distance between the second counting matrix and the target cluster centroid is less than a third threshold, the log data to be detected corresponding to the second counting matrix is ​​determined to be abnormal log data.

[0119] It should be noted that the sample log data includes log data generated during normal module operation and log data containing fault information. When performing hierarchical clustering on the clustering model using the sample log data, the first cluster centroid of the log data generated during normal operation and the second cluster centroid of the log data containing fault information can be obtained respectively. The first cluster centroid and the second cluster centroid together constitute the aforementioned target cluster centroid.

[0120] The log anomaly detection method provided by this invention converts the log data to be detected into a second counting matrix and identifies it through a log anomaly detection model. This method can locate hidden fault information in the signal system and facilitate subsequent fault handling by professional technicians.

[0121] The training apparatus for the log anomaly detection model provided by the present invention will be described below. The training apparatus for the log anomaly detection model described below can be referred to in correspondence with the training method for the log anomaly detection model described above.

[0122] Figure 4 This is a schematic diagram of the training device for the log anomaly detection model provided by the present invention, as shown below. Figure 4 As shown, the training device for the log anomaly detection model includes: an acquisition module 410, a parsing module 420, a feature extraction module 430, and a clustering module 440.

[0123] The acquisition module 410 is used to acquire various sample log data generated by the signal system;

[0124] The parsing module 420 is used to parse multiple sample log data based on event templates to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed sample log data corresponding to the target parameter. The event template is determined based on unstructured log data, and the target parameter is used to represent the category information of the sample log data.

[0125] The feature extraction module 430 is used to sort multiple log templates based on the edit distance between multiple log templates and the word vector similarity between multiple parsed sample log data to obtain a second template sequence, and add the first keyword and the statistical information of the target log template to the second template sequence to obtain a first counting matrix. The target log template is the log template that stores the most parsed sample log data among multiple log templates. The first keyword is obtained based on the printing parameters in multiple sample log data.

[0126] Clustering module 440 is used to input the first counting matrix into the clustering model for hierarchical clustering training, and obtain the log anomaly detection model after training.

[0127] Figure 5An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a training method for a log anomaly detection model. This method includes: acquiring multiple sample log data generated by a signal system; parsing the multiple sample log data based on event templates to obtain a first template sequence; sorting the multiple log templates based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data, and numbering the sorted multiple log templates to obtain a second template sequence; adding a first keyword and statistical information of the target log template to the second template sequence to obtain a first counting matrix; inputting the first counting matrix into a clustering model for hierarchical clustering training, and obtaining a log anomaly detection model after training. Alternatively, a log anomaly detection method can be implemented, which includes: acquiring log data to be detected; parsing the log data to be detected based on event templates to obtain a first template sequence; sorting multiple log templates based on the edit distance between multiple log templates and the word vector similarity between multiple parsed log data to be detected to obtain a second template sequence, and adding a first keyword and statistical information of the target log template to the second template sequence to obtain a second counting matrix; clustering the second counting matrix based on a log anomaly detection model to obtain a target clustering distance, and determining that the log data to be detected is abnormal log data if the target clustering distance is less than a third threshold.

[0128] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the log anomaly detection model provided by the above methods. The method includes: acquiring multiple sample log data generated by a signal system; parsing the multiple sample log data based on event templates to obtain a first template sequence; sorting the multiple log templates based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence, and adding a first keyword and statistical information of the target log template to the second template sequence to obtain a first counting matrix; inputting the first counting matrix into a clustering model for hierarchical clustering training, and obtaining a log anomaly detection model after training. Alternatively, a log anomaly detection method can be implemented, which includes: acquiring log data to be detected; parsing the log data to be detected based on event templates to obtain a first template sequence; sorting multiple log templates based on the edit distance between multiple log templates and the word vector similarity between multiple parsed log data to be detected, and numbering the sorted multiple log templates to obtain a second template sequence; adding a first keyword and statistical information of the target log template to the second template sequence to obtain a second counting matrix; clustering the second counting matrix based on a log anomaly detection model to obtain a target clustering distance, and determining that the log data to be detected is abnormal log data if the target clustering distance is less than a third threshold.

[0130] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method for the log anomaly detection model provided by the above methods. The method includes: acquiring multiple sample log data generated by a signal system; parsing the multiple sample log data based on event templates to obtain a first template sequence; sorting the multiple log templates based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence, and adding a first keyword and statistical information of the target log template to the second template sequence to obtain a first counting matrix; inputting the first counting matrix into a clustering model for hierarchical clustering training, and obtaining a log anomaly detection model after training. Alternatively, a log anomaly detection method can be implemented, which includes: acquiring log data to be detected; parsing the log data to be detected based on event templates to obtain a first template sequence; sorting multiple log templates based on the edit distance between multiple log templates and the word vector similarity between multiple parsed log data to be detected, and numbering the sorted multiple log templates to obtain a second template sequence; adding a first keyword and statistical information of the target log template to the second template sequence to obtain a second counting matrix; clustering the second counting matrix based on a log anomaly detection model to obtain a target clustering distance, and determining that the log data to be detected is abnormal log data if the target clustering distance is less than a third threshold.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a log anomaly detection model, characterized in that, include: Acquire various sample log data generated by the signal system; The various sample log data are parsed based on event templates to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed sample log data corresponding to the target parameter. The event template is determined based on unstructured log data. The target parameter is used to represent the category information of the sample log data. The log templates are sorted based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence. The three words with the highest TF-IDF values ​​are obtained by calculating the TF-IDF values ​​of the first template sequence as the first keywords. The first keywords are converted into numbers using the correspondence between words and sequence numbers in the corpus. The converted first keywords and the statistical information of the target log template are added to the second template sequence to obtain a first counting matrix. The target log template is the log template that stores the most parsed sample log data among the multiple log templates. The first keywords are obtained based on the printing parameters in the multiple sample log data. The first counting matrix is ​​input into the clustering model for hierarchical clustering training, and the log anomaly detection model is obtained after training.

2. The training method for the log anomaly detection model according to claim 1, characterized in that, The step of inputting the first counting matrix into a clustering model for hierarchical clustering training, and obtaining a log anomaly detection model after training, includes: The first counting matrix is ​​input into a clustering model for clustering, resulting in multiple sample clusters; The target distance between any two sample clusters in the plurality of sample clusters is obtained based on the mean clustering formula. When the target distance is less than a first threshold, the two sample clusters are merged to obtain a fused cluster. The log anomaly detection model is obtained when the target distance is greater than the first threshold or the number of fusion clusters is less than the second threshold.

3. The training method for the log anomaly detection model according to claim 1, characterized in that, The step of inputting the first counting matrix into the clustering model for hierarchical clustering training, and obtaining the log anomaly detection model after training, further includes: The counting matrix is ​​normalized and vectorized to obtain the counting vector; The counting vector is input into the clustering model for hierarchical clustering training, and the log anomaly detection model is obtained after training.

4. The training method for the log anomaly detection model according to claim 1, characterized in that, The steps for determining the event template include: Obtain the token object from the unstructured log data; The event template is obtained by classifying the unstructured log data based on the token object and the second keyword in the unstructured log data.

5. The training method for the log anomaly detection model according to claim 2, characterized in that, The mean clustering formula includes: ， Among them, C i and C j They are any two sample clusters from the plurality of sample clusters. C i The average value, C j The average value.

6. A method for detecting log anomalies, characterized in that, include: Obtain the log data to be tested; The log data to be detected is parsed based on the event template to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed log data to be detected corresponding to the target parameter. The event template is determined based on unstructured log data. The target parameter is used to represent the category information of the log data to be detected. The log templates are sorted based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence. The three words with the highest TF-IDF values ​​are obtained by calculating the TF-IDF values ​​of the first template sequence as the first keywords. The first keywords are converted into numbers using the correspondence between words and sequence numbers in the corpus. The converted first keywords and the statistical information of the target log template are added to the second template sequence to obtain a first counting matrix. The target log template is the log template that stores the most parsed sample log data among the multiple log templates. The first keywords are obtained based on the printing parameters in the multiple sample log data. Based on the log anomaly detection model as described in any one of claims 1 to 5, the second counting matrix is ​​clustered to obtain the target clustering distance, and if the target clustering distance is less than a third threshold, the log data to be detected is determined to be abnormal log data.

7. A training device for a log anomaly detection model, characterized in that, include: The acquisition module is used to acquire various sample log data generated by the signal system; The parsing module is used to parse the various sample log data based on event templates to obtain a first template sequence. The first template sequence includes multiple log templates. Each log template is used to store the parsed sample log data corresponding to the target parameter. The event template is determined based on unstructured log data. The target parameter is used to represent the category information of the sample log data. The feature extraction module is used to sort the multiple log templates based on the edit distance between the multiple log templates and the word vector similarity between the multiple parsed sample log data to obtain a second template sequence. By calculating the TF-IDF value of the first template sequence, the three words with the highest TF-IDF value are obtained as the first keywords. The first keywords are converted into numbers using the correspondence between words and numbers in the corpus. The converted first keywords and the statistical information of the target log template are added to the second template sequence to obtain a first counting matrix. The target log template is the log template that stores the most parsed sample log data among the multiple log templates. The first keywords are obtained based on the printing parameters in the multiple sample log data. The clustering module is used to input the first counting matrix into the clustering model for hierarchical clustering training, and to obtain the log anomaly detection model after training.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method of the log anomaly detection model as described in any one of claims 1 to 5 or the log anomaly detection method as described in claim 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the log anomaly detection model as described in any one of claims 1 to 5 or the log anomaly detection method as described in claim 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the log anomaly detection model as described in any one of claims 1 to 5 or the log anomaly detection method as described in claim 6.

Images