Method and apparatus for determining abnormal logs

Abstract

The application provides a kind of determination method and device of abnormal log.The method comprises: obtaining real-time log sequence according to first preset time length;Real-time log sequence is input into log quantity anomaly detection model, and the quantity anomaly detection of the log quantity of real-time log sequence is carried out;In the case where the log quantity of real-time log sequence is abnormal, real-time log sequence is input into classification anomaly detection model, and the feature anomaly detection of real-time log sequence is carried out;In the case where the feature anomaly detection result of real-time log sequence is abnormal, the log data of second preset time length before current time is input into time sequence anomaly detection model to carry out log mode quantity distribution judgment, and determine abnormal log.The determination method and device of abnormal log provided by the application, by determining the real-time log sequence of quantity anomaly first, then according to classification anomaly detection model and time sequence anomaly detection model, the detection of abnormal log is carried out, and the detection speed and accuracy of log are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication technology, in particular to a method and device for determining abnormal logs. BACKGROUND

[0002] The existing log abnormality analysis methods mainly include a log pattern sequence prediction detection method represented by DeepLog, a direct abnormality detection analysis method for keywords such as log levels in logs, and a LogRobust abnormality detection analysis method based on log semantics.

[0003] The existing method DeepLog uses an LSTM (Long Short-Term Memory) algorithm to predict log pattern sequences, which requires that the log pattern runs regularly, otherwise the effect is poor. Some log patterns are normal but appear less frequently or irregularly and are easily detected as abnormal. The supervised classification abnormality detection method for keywords such as log levels has great limitations, and the method is simple and crude, and the effect is not ideal. The LogRobust analysis is based on context analysis and needs original logs for analysis, and the performance is poor. In actual production, the number of logs is very large, and it is difficult to meet the production requirements.

[0004] Therefore, it is of great significance to propose a method that can detect abnormal logs for regular logs or irregular logs, and has fast detection speed and good detection effect. SUMMARY

[0005] The present application provides a method and device for determining abnormal logs to solve the technical problems of the prior art that the existing method for determining abnormal logs cannot simultaneously detect regular or irregular logs quickly and has poor detection effect.

[0006] In a first aspect, the present application provides a method for determining abnormal logs, comprising:

[0007] obtaining a real-time log sequence according to a first preset time length;

[0008] inputting the real-time log sequence into a log quantity abnormality detection model to perform quantity abnormality detection on the log quantity of the real-time log sequence;

[0009] in the case of log quantity abnormality of the real-time log sequence, inputting the real-time log sequence into a classification abnormality detection model to perform feature abnormality detection on the real-time log sequence;

[0010] in the case of abnormal feature abnormality detection result of the real-time log sequence, inputting log data of a second preset time length before the current time into a time sequence abnormality detection model to perform log pattern quantity distribution judgment, and determining abnormal logs.

[0011] The second preset time length is greater than or equal to the first preset time length.

[0012] In one embodiment, the number anomaly detection on the number of logs in the real-time log sequence comprises:

[0013] The historical log sequence corresponding to the time of the real-time log sequence is obtained according to the first preset time length;

[0014] The historical log sequence is input into a log number anomaly detection model for log number prediction;

[0015] The difference between the number of logs in the real-time log sequence and the log number prediction value is determined, and if the difference is greater than a preset threshold, it is determined that the number of logs in the real-time log sequence is abnormal.

[0016] In one embodiment, the feature anomaly detection on the real-time log sequence comprises:

[0017] A log mode list is determined from the real-time log sequence based on a solidified drain3 mode extraction algorithm, feature extraction is performed on the log mode list, and a feature list is obtained;

[0018] The log feature list is filtered according to a pre-defined log feature whitelist;

[0019] The filtered log feature list is input into a classification anomaly detection model for feature anomaly detection.

[0020] In one embodiment, the log data of the second preset time length before the current time is input into a time series anomaly detection model for log mode number distribution judgment to determine the abnormal log, which comprises:

[0021] The log data of the second preset time length is obtained as a first log sequence according to the first preset time length;

[0022] According to the Cauchy distribution, the cumulative distribution function probability of the first log sequence is determined to be within a preset range as an abnormal log.

[0023] In one embodiment, after the log data of the second preset time length is obtained as a first log sequence according to the first preset time length, it further comprises:

[0024] The number of log modes of the first log sequence is obtained, and low-pass filtering processing is performed on the number of log modes;

[0025] The number of log modes after low-pass filtering processing is determined as a log mode number error;

[0026] The median of the log mode number error and the absolute deviation of the median of the log mode number error are determined;

[0027] The cumulative distribution function probability is calculated according to the median and the median absolute deviation.

[0028] In one embodiment, the log quantity anomaly detection model is trained based on the following steps:

[0029] The first historical log sequence is obtained according to the first preset time length;

[0030] The first historical log sequence is input into the time series model Prophet for training to obtain the log quantity anomaly detection model.

[0031] In one embodiment, the classification anomaly detection model is trained based on the following steps:

[0032] The second historical log sequence is obtained according to the first preset time length;

[0033] The log mode list is determined from the second historical log sequence based on the solidified drain3 mode extraction algorithm, feature extraction is performed on the log mode list, and a feature list is obtained;

[0034] The log feature list is screened according to a pre-defined log feature whitelist;

[0035] The screened log features are labeled;

[0036] The labeled log features are input into a support vector machine (SVM) algorithm for training to obtain the classification anomaly detection model;

[0037] The solidified drain3 mode extraction algorithm is trained based on the second historical log sequence.

[0038] In a second aspect, the present application further provides a device for determining abnormal logs, comprising:

[0039] A data acquisition module is configured to obtain a real-time log sequence according to a first preset time length;

[0040] An abnormal quantity detection module is configured to input the real-time log sequence into a log quantity anomaly detection model to perform quantity anomaly detection on the log quantity of the real-time log sequence;

[0041] A classification anomaly detection module is configured to, in the case of log quantity anomaly of the real-time log sequence, input the real-time log sequence into a classification anomaly detection model to perform feature anomaly detection on the real-time log sequence;

[0042] A time sequence anomaly detection module is configured to, in the case of abnormal feature anomaly detection result of the real-time log sequence, input log data of a second preset time length before the current time into a time sequence anomaly detection model to perform log mode quantity distribution judgment and determine abnormal logs.

[0043] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for determining an abnormal log according to any one of the above aspects when executing the computer program.

[0044] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for determining an abnormal log according to any one of the above aspects.

[0045] The method and device for determining an abnormal log, the electronic device, and the storage medium provided by the present application can preliminarily determine whether the real-time log sequence of a preset time length is abnormal according to a log quantity abnormality detection model. Based on the log quantity abnormality determination, abnormal log detection can be performed on logs with or without rules, and the real-time logs can be preliminarily and quickly screened. The real-time log sequence preliminarily screened according to the quantity abnormality is subjected to subsequent accurate abnormal log detection by a classification abnormality detection model and a time sequence abnormality detection model, so as to further improve the accuracy of log detection. The entire detection process improves the detection speed of log abnormality detection while ensuring the accuracy of detection. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0047] Figure 1 The flowchart of the method for determining an abnormal log provided by the present application is shown in the figure.

[0048] Figure 2 The overall flow processing architecture diagram of the method for determining an abnormal log provided by an embodiment of the present application is shown in the figure.

[0049] Figure 3 The basic principle diagram of the support vector machine algorithm provided by an embodiment of the present application is shown in the figure.

[0050] Figure 4 The structure diagram of the device for determining an abnormal log provided by the present application is shown in the figure.

[0051] Figure 5 The structure diagram of the electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described below in a clear and complete manner with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0053] Figure 1 A flowchart of the abnormal log determination method provided by the present application is shown in FIG. 1. Referring to FIG. 1, Figure 1 The abnormal log determination method provided by the present application can include the following steps.

[0054] S110, acquiring a real-time log sequence according to a first preset time length;

[0055] S120, inputting the real-time log sequence into a log quantity abnormality detection model to perform quantity abnormality detection on the log quantity of the real-time log sequence;

[0056] S130, in the case of log quantity abnormality of the real-time log sequence, inputting the real-time log sequence into a classification abnormality detection model to perform feature abnormality detection on the real-time log sequence;

[0057] S140, in the case of abnormal feature abnormality detection result of the real-time log sequence, inputting log data of a second preset time length before the current time into a time sequence abnormality detection model to perform log mode quantity distribution judgment, and determining abnormal logs;

[0058] The second preset time length is greater than or equal to the first preset time length.

[0059] The execution subject of the abnormal log determination method provided by the present application can be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., which are not limited in the present application.

[0060] The technical scheme of the present application is described in detail below by taking the method for determining abnormal logs executed by a computer as an example.

[0061] It should be noted that in actual production, log analysis is one of the important fault diagnosis methods. Compared with logs generated in normal operation state, abnormal logs generated in fault condition will increase in quantity. In addition, key abnormal information is often printed in the logs to help operation and maintenance personnel locate. By determining abnormal logs, the positioning and analysis of faults can be realized.

[0062] The overall process of the technical scheme of the present application consists of two parts: offline training and online detection. For details, refer to Figure 2 The overall process is shown in the processing architecture diagram.

[0063] In step S110, a real-time log sequence is obtained according to a first preset time length.

[0064] Optionally, the first preset time length can be a time length unit such as minute, hour, day, etc. However, in the actual log abnormality detection process, in order to ensure real-time performance and detection efficiency, minute can be selected as the time length of the real-time log sequence.

[0065] In step S120, the real-time log sequence obtained in step S110 is input into a log quantity abnormality detection model to obtain the log quantity in the time period of the first preset time length. The log quantity abnormality detection model is used to detect the quantity of the real-time log sequence, to determine whether the log quantity of the real-time log sequence is normal, and to output the real-time log sequence for the next step of judgment.

[0066] It can be understood that the quantity of logs at the current time under normal state changes little compared with the quantity of historical logs at the same time. When a fault occurs at the current time, a large number of fault-related abnormal logs will be generated. By using the log quantity abnormality detection model to determine the quantity of real-time logs, the real-time logs can be preliminarily and quickly screened.

[0067] In step S130, the quantity of the real-time log sequence obtained in step S120 is determined according to the log quantity abnormality detection model. If the determination result shows that the quantity of the current real-time log sequence is normal, the subsequent detection process is ended, and the real-time log sequence at the next time is obtained for log quantity detection in step S120. If the determination result shows that the quantity of the current real-time log sequence is abnormal, the current real-time log sequence is subjected to the next step of detection, i.e., the current real-time log sequence is input into a classification abnormality detection model, the log features of the real-time log sequence are extracted, and the log features are subjected to further feature abnormality detection.

[0068] In step S140, the log features of the real-time log sequence are subjected to feature anomaly detection according to the classification anomaly detection model in step S130. If the judgment result shows that the feature anomaly detection result of the real-time log sequence at the time is normal, the subsequent detection process is ended, and the detection process is continued by acquiring the real-time log sequence at the next time. If the judgment result shows that the feature anomaly detection result of the current real-time log sequence is abnormal, the current real-time log sequence is subjected to the detection of the next step, that is, the log data of the second preset time length before the current time is input into the time sequence anomaly detection model, the log anomaly is judged according to the distribution of the number of log modes, and the abnormal log is further determined.

[0069] Optionally, the second preset time length can be selected as the time length of time units such as hours or days, for example, the log data of two hours before the current time is selected to input the time sequence anomaly detection model for detection.

[0070] The determination method of abnormal logs provided by the application can preliminarily judge whether the number of real-time log sequences is abnormal according to the log number anomaly detection model by acquiring the real-time log sequence of a preset time length. The log anomaly judgment based on the number of logs can detect abnormal logs for regular or irregular logs, and can realize the preliminary and rapid screening of real-time logs. The number abnormal real-time log sequences screened preliminarily are subjected to the accurate detection of subsequent abnormal logs by the classification anomaly detection model and the time sequence anomaly detection model, and the accuracy of log detection is further improved. The entire detection process improves the detection speed of log anomaly detection under the premise of ensuring the accuracy of detection.

[0071] In one embodiment, the number anomaly detection of the number of real-time log sequences includes: acquiring a historical log sequence corresponding to the time of the real-time log sequence according to a first preset time length; inputting the historical log sequence into a log number anomaly detection model to perform log number prediction; determining the difference between the number of logs of the real-time log sequence and the log number prediction value, and if the difference is greater than a preset threshold, determining that the number of logs of the real-time log sequence is abnormal.

[0072] Optionally, the real-time log sequence at the current time such as 10:01 can be acquired by minutes. The length of the real-time log sequence is one minute, that is, the real-time log in the time period from 10:01 to 10:02. At the same time, the historical log sequence at the same time of 10:01 the day before the current time is acquired by minutes. The historical log sequence is input into the log number anomaly detection model to perform historical log sequence number prediction. The number of logs of the one-minute real-time log sequence at the current time 10:01 is acquired. The difference between the predicted number of the historical log sequence and the number of logs of the real-time log sequence at the current time is calculated, and if the difference is greater than a preset threshold, it is considered that the number of logs at the time is abnormal.

[0073] The abnormal log determination method provided by the application determines whether the log quantity of the current real-time log sequence is normal according to the difference value, so as to determine whether the current real-time log sequence is an abnormal log sequence, thereby achieving preliminary and rapid screening of real-time logs.

[0074] In one embodiment, the feature anomaly detection on the real-time log sequence comprises: determining a log mode list from the real-time log sequence based on the solidified drain3 mode extraction algorithm, performing feature extraction on the log mode list to obtain a feature list; screening the log feature list according to a pre-defined log feature whitelist; and inputting the screened log feature list into a classification anomaly detection model for feature anomaly detection.

[0075] Optionally, historical log data of a time interval greater than one day is obtained, the historical data is input into the drain3 mode extraction algorithm, the drain3 mode extraction algorithm is trained, and the log extraction mode is solidified. According to the solidified drain3 mode extraction algorithm, a log mode list is determined from the real-time log sequence, and a feature list of J features is obtained from the log mode list. According to the W log feature white lists of the features pre-defined by the user, screening is performed, and the remaining J-W remaining log features are input into a classification anomaly detection model for feature anomaly detection.

[0076] It can be understood that the log features are further judged for abnormality according to the classification anomaly detection model, if the detection result of the feature abnormality judgment of the real-time log sequence is true, subsequent detection is continued, and if the detection result is false, the detection is terminated.

[0077] The abnormal log determination method provided by the application further judges the log features for abnormality through the classification anomaly detection model, further determines the abnormality of the log according to the judgment result in the case of preliminary judgment of log quantity abnormality, and ensures the accuracy of the detection on the premise of improving the detection speed of the log abnormality detection.

[0078] In one embodiment, the log data of a second preset time length before the current time is input into a time sequence anomaly detection model for log mode quantity distribution judgment, and the abnormal log is determined by comprising: obtaining a first log sequence from the log data of the second preset time length according to a first preset time length; and determining, according to the Cauchy distribution, that the cumulative distribution function probability of the first log sequence is within a preset range as an abnormal log.

[0079] Optionally, log data of two hours before the current time can be acquired. For example, two hours of log data are aggregated by minute, one historical log data sequence is aggregated every minute, and 120 log sequence sets are obtained. The number of logs in each log sequence of the log sequence set is acquired, and the distribution function probability of the number of log modes of each log sequence is calculated according to the Cauchy distribution. If it is judged that the distribution function probability is within a pre-set range, it is judged as an abnormal log, thereby locating a specific abnormal log, and outputting the abnormal log. The number of log modes can include info type information, debug type information, business type markers, and the like, which are extracted according to the solidified drain3 mode extraction algorithm.

[0080] It can be understood that the abnormality detection herein is based on log data of the last two hours before the current time, and is not periodic. If a fault occurs, the current log will increase in quantity. Therefore, the probability calculation method according to the Cauchy distribution has good detection effect.

[0081] The abnormal log determination method provided by the application determines the abnormality of historical data of a current log sequence at a time according to the Cauchy distribution, further determines the abnormality of the log according to the judgment result in the case of preliminary judgment of log quantity abnormality, and ensures the accuracy of detection on the premise of improving the detection speed of log abnormality detection.

[0082] In one embodiment, after the log data of the second preset time length is acquired to obtain the first log sequence according to the first preset time length, the method further includes: acquiring the number of log modes of the first log sequence, performing low-pass filtering processing on the number of log modes; determining the error of the number of log modes after the low-pass filtering processing and the number of log modes of the first log sequence; determining the median of the number of log modes and the absolute deviation of the median of the number of log modes; and calculating the cumulative distribution function probability according to the median and the absolute deviation of the median.

[0083] Optionally, log data of two hours before the current time can be acquired. Two hours of log data are aggregated by minute, one historical log data sequence is aggregated every minute, and 120 log sequence sets N are obtained. The current time log sequence is denoted as n. The number of log modes of the current time log sequence is subjected to low-pass filtering processing:

[0084] X n =X n-1 *α+(1-α)*Y n (1)

[0085] X n denotes the smoothed value of the number of log modes in the current time log sequence; and X n-1Y represents the smoothed value indicating the number of log patterns in the log sequence at the previous time step; n This represents the number of log patterns in the log sequence at the current time; α is the smoothing coefficient, which can be a constant of 0.7; at time 0, X n The value is 0.

[0086] The error between the number of log patterns after low-pass filtering and the number of log patterns in the first log sequence is further determined using formula (2):

[0087] Err n =Y n -X n (2)

[0088] Among them, Err n This represents the error between the number of log patterns in the log sequence at the current time and the smoothed value of the number of log patterns in the log sequence at the current time.

[0089] The median of the log pattern quantity error is further calculated using formula (3):

[0090] Meadian_=meadian 1<＝n<＝N (Err n (3)

[0091] Where Meadian_ represents the median of the error; meadian means to calculate the median.

[0092] After calculating the median of the log pattern quantity error, the median absolute deviation of the log pattern quantity error is further calculated using formula (4):

[0093] MAD = meadian 1<＝n<＝N (|Err n -Meadian_|) (4)

[0094] MAD represents the median absolute deviation of the log pattern quantity error.

[0095] Calculate the cumulative distribution function probability in the Cauchy distribution based on the median of the log pattern quantity error and the median absolute deviation of the log pattern quantity error:

[0096]

[0097] CDF(Y) n ) represents the cumulative distribution function probability.

[0098] Optionally, if the calculated cumulative distribution function probability CDF(Y) n) greater than or equal to 0.95 or less than or equal to 0.05, the current log sequence n is determined as an abnormal log sequence. Otherwise, it is determined as a normal log sequence.

[0099] The abnormal log determination method provided by the application determines the abnormality of the historical data of the current log sequence at a time point one period of time ago according to the Cauchy distribution, further determines the abnormality of the log according to the result in the case of preliminary determination of the abnormality of the log quantity, and ensures the accuracy of the detection under the premise of improving the detection speed of the log abnormality detection.

[0100] In one embodiment, the log quantity abnormality detection model is obtained by training based on the following steps: obtaining a first historical log sequence according to a first preset time length; inputting the first historical log sequence into a time series model Prophet for training to obtain the log quantity abnormality detection model.

[0101] Optionally, historical log data of a time interval of at least one day is selected. The historical log data is obtained as a historical log sequence by minute, and the log quantity of the historical log sequence is input into the time series model Prophet for training. The log quantity of the input historical log sequence is fitted to obtain the log quantity abnormality detection model.

[0102] It should be noted that Prophet is a data prediction tool based on Python and R language opened by Facebook. Prophet is more user-friendly than existing prediction tools.

[0103] The abnormal log determination method provided by the application trains the time series model Prophet according to the historical log data to construct a log quantity abnormality detection model. In the process of real-time detection, the log quantity abnormality detection model is used to obtain the difference between the log quantity value of the real-time log sequence and the predicted log quantity value of the historical log sequence at the corresponding time point, and the difference is used to determine whether the log quantity of the current real-time log sequence is normal, so as to determine whether the current real-time log sequence is an abnormal log sequence, so as to achieve the preliminary and rapid screening of the real-time log.

[0104] In one embodiment, the classification anomaly detection model is trained based on the following steps: acquiring a second historical log sequence for a first preset duration; determining a log pattern list from the second historical log sequence based on a fixed drain3 pattern extraction algorithm, extracting features from the log pattern list to obtain a feature list; filtering the log feature list according to a predefined log feature whitelist; labeling the filtered log features; and inputting the labeled log features into a support vector machine (SVM) algorithm for training to obtain the classification anomaly detection model; wherein the fixed drain3 pattern extraction algorithm is trained based on the second historical log sequence.

[0105] Optionally, historical log data with a time interval longer than one day is acquired. This historical data is then fed into the drain3 pattern extraction algorithm to train the algorithm and solidify the log extraction patterns. Next, the historical log data is re-extracted using the solidified log model to create a list of historical log patterns, assuming the extracted log pattern list contains M patterns. Based on the extracted log pattern list, K-1 log patterns are selected as K-1 features for the classification anomaly detection algorithm, where K > 2. The remaining M-K+1 patterns are treated as an "Other" feature, which, together with the previous K-1 features, forms K features. The number of K features per minute is aggregated and counted along two dimensions: log pattern and minute. If a log pattern feature does not appear within a certain minute interval, the count of that feature is set to 0.

[0106] Based on a user-defined whitelist of W log features, K features are filtered. The remaining KW features and their quantities are then manually labeled in a two-dimensional matrix. Fault time intervals are marked as 1, and normal time intervals are marked as 0, resulting in the labeling matrix shown in Table 1. The labeled matrix is ​​then input into a Support Vector Machine (SVM) algorithm for training, yielding a classification anomaly detection model.

[0107] Table 1 Log Feature Marker Table

[0108]

[0109] It should be noted that SVM is a classic machine learning classification algorithm. The basic principle diagram of the algorithm is shown below. Figure 3The SVM algorithm is used to distinguish two classes of samples by finding support vectors and maximizing the margin. The data of the K-dimensional feature vector is labeled as abnormal and non-abnormal. Here, the feature vector dimension is high and is not necessarily linearly separable. Therefore, the support vector machine using kernel trick can well solve this problem. The basic idea is to use a transformation to map the data in the original space to a new space, such as a higher-dimensional space or an infinite-dimensional space, and then learn a model from the data in the new space using a linear method.

[0110] The abnormal log determination method provided by the application further judges the log features through the classification anomaly detection model, without the need for a large number of labels on the original real-time log. Only a small number of labels are needed on the patterned log during model training, that is, detection is performed through the trained classification detection model, which improves the detection efficiency. At the same time, the model is trained using the artificial labeling method to manually label the abnormal log, and the subsequent real-time detection is also based on the abnormal log calculated by the artificial labeling, which has good interpretability.

[0111] The application further provides an abnormal log determination device, which can be correspondingly referred to the abnormal log determination method described above.

[0112] Figure 4 The structure diagram of the abnormal log determination device provided by the application is shown in Figure 4 The device comprises:

[0113] The data acquisition module 410 is configured to acquire a real-time log sequence according to a first preset time length.

[0114] The abnormal number detection module 420 is configured to input the real-time log sequence into a log number anomaly detection model to perform number anomaly detection on the log number of the real-time log sequence.

[0115] The classification anomaly detection module 430 is configured to input the real-time log sequence into a classification anomaly detection model to perform feature anomaly detection on the real-time log sequence in the case of log number anomaly of the real-time log sequence.

[0116] The time sequence anomaly detection module 440 is configured to input log data of a second preset time length before the current time into a time sequence anomaly detection model to perform log mode number distribution judgment and determine an abnormal log in the case of abnormal feature anomaly detection result of the real-time log sequence.

[0117] The application provides an abnormal log determination device. The device obtains a real-time log sequence of a preset time length, and determines whether the real-time log sequence is abnormal according to a log quantity abnormality detection model. The log quantity abnormality detection model can be used to detect abnormal logs in a regular or irregular manner, and can be used to preliminarily and rapidly screen real-time logs. The real-time log sequence screened preliminarily and abnormally in quantity is subjected to subsequent accurate detection of abnormal logs by a classification abnormality detection model and a time sequence abnormality detection model, so that the accuracy of log detection is further improved. The detection process improves the detection speed of log abnormality detection and ensures the accuracy of detection.

[0118] In one embodiment, the abnormal quantity detection module 420 is specifically configured to:

[0119] The quantity abnormality detection on the log quantity of the real-time log sequence includes:

[0120] The historical log sequence corresponding to the time of the real-time log sequence is obtained according to the first preset time length;

[0121] The historical log sequence is input into the log quantity abnormality detection model to perform log quantity prediction;

[0122] The difference between the log quantity of the real-time log sequence and the predicted log quantity is determined, and if the difference is greater than a preset threshold, it is determined that the log quantity of the real-time log sequence is abnormal.

[0123] In one embodiment, the classification abnormality detection module 430 is specifically configured to:

[0124] The feature abnormality detection on the real-time log sequence includes:

[0125] The log mode list is determined from the real-time log sequence based on the solidified drain3 mode extraction algorithm, the feature extraction is performed on the log mode list, and the feature list is obtained;

[0126] The log feature list is screened according to a pre-defined log feature white list;

[0127] The screened log feature list is input into the classification abnormality detection model to perform feature abnormality detection.

[0128] In one embodiment, the time sequence abnormality detection module 440 is specifically configured to:

[0129] The log data of the second preset time length before the current time is input into the time sequence abnormality detection model to perform log mode quantity distribution judgment, and the abnormal log includes:

[0130] The first log sequence is obtained from the log data of the second preset time length according to the first preset time length;

[0131] The cumulative distribution function probability of the first log sequence determined according to the Cauchy distribution is within a preset range, and the log is an abnormal log.

[0132] In one embodiment, the timing anomaly detection module 440 is further specifically used for:

[0133] After the log data of the second preset time length is used to obtain the first log sequence in the first preset time length, the method further includes:

[0134] The number of log patterns of the first log sequence is obtained, and low-pass filtering processing is performed on the number of log patterns.

[0135] The number of log patterns after low-pass filtering processing is determined to be an error with the number of log patterns of the first log sequence.

[0136] The median of the number of log patterns error and the absolute deviation of the median of the number of log patterns error are determined.

[0137] The cumulative distribution function probability is calculated according to the median and the absolute deviation of the median.

[0138] In one embodiment, the anomaly number detection module 420 is further specifically used for:

[0139] The log number anomaly detection model is trained based on the following steps:

[0140] The first historical log sequence is obtained in the first preset time length.

[0141] The first historical log sequence is input into the time series model Prophet for training to obtain the log number anomaly detection model.

[0142] In one embodiment, the classification anomaly detection module 430 is further specifically used for:

[0143] The classification anomaly detection model is trained based on the following steps:

[0144] The second historical log sequence is obtained in the first preset time length.

[0145] Based on the solidified drain3 mode extraction algorithm, a log pattern list is determined from the second historical log sequence, and a feature list is obtained by performing feature extraction on the log pattern list.

[0146] According to a pre-defined log feature whitelist, the log feature list is screened.

[0147] The screened log features are labeled.

[0148] The labeled log features are input into a support vector machine (SVM) algorithm for training to obtain the classification anomaly detection model.

[0149] The solidified drain 3 mode extraction algorithm is obtained by training the drain 3 mode extraction algorithm based on the second historical log sequence.

[0150] The application also provides an electronic device, such as Figure 5 As shown in the figure, the electronic device can include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 can communicate with each other through the communication bus 540. The processor 510 can call the logical instructions in the memory 530 to execute the steps of the abnormal log determination method, for example, including:

[0151] obtaining a real-time log sequence according to a first preset time length;

[0152] inputting the real-time log sequence into a log quantity anomaly detection model to perform quantity anomaly detection on the log quantity of the real-time log sequence;

[0153] in the case of log quantity anomaly of the real-time log sequence, inputting the real-time log sequence into a classification anomaly detection model to perform feature anomaly detection on the real-time log sequence;

[0154] in the case of feature anomaly detection result anomaly of the real-time log sequence, inputting log data of a second preset time length before the current time into a time sequence anomaly detection model to perform log mode quantity distribution judgment and determine abnormal logs;

[0155] wherein the second preset time length is greater than or equal to the first preset time length.

[0156] In addition, the logical instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0157] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the steps of the method for determining abnormal logs provided by each method embodiment described above, for example comprising:

[0158] obtaining a real-time log sequence according to a first preset time length;

[0159] inputting the real-time log sequence into a log quantity anomaly detection model to perform quantity anomaly detection on the log quantity of the real-time log sequence;

[0160] in a case where the log quantity of the real-time log sequence is abnormal, inputting the real-time log sequence into a classification anomaly detection model to perform feature anomaly detection on the real-time log sequence;

[0161] in a case where the feature anomaly detection result of the real-time log sequence is abnormal, inputting log data of a second preset time length before the current time into a time series anomaly detection model to perform log mode quantity distribution judgment, and determining abnormal logs;

[0162] wherein the second preset time length is greater than or equal to the first preset time length.

[0163] In another aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method for determining abnormal logs provided by each method embodiment described above, for example comprising:

[0164] obtaining a real-time log sequence according to a first preset time length;

[0165] inputting the real-time log sequence into a log quantity anomaly detection model to perform quantity anomaly detection on the log quantity of the real-time log sequence;

[0166] in a case where the log quantity of the real-time log sequence is abnormal, inputting the real-time log sequence into a classification anomaly detection model to perform feature anomaly detection on the real-time log sequence;

[0167] in a case where the feature anomaly detection result of the real-time log sequence is abnormal, inputting log data of a second preset time length before the current time into a time series anomaly detection model to perform log mode quantity distribution judgment, and determining abnormal logs;

[0168] wherein the second preset time length is greater than or equal to the first preset time length.

[0169] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0170] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0171] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of determining an abnormal log, characterized by, The method comprises the following steps: obtaining a real-time log sequence according to a first preset time length; inputting the real-time log sequence into a log quantity anomaly detection model to perform quantity anomaly detection on the log quantity of the real-time log sequence; in the case of log quantity anomaly of the real-time log sequence, inputting the real-time log sequence into a classification anomaly detection model to perform feature anomaly detection on the real-time log sequence; in the case of abnormal feature anomaly detection result of the real-time log sequence, obtaining a first log sequence from log data of a second preset time length according to a first preset time length; and determining, according to Cauchy distribution, that a cumulative distribution function probability of the first log sequence is an abnormal log in a preset range; wherein the second preset time length is greater than or equal to the first preset time length; a calculation formula of the cumulative distribution function probability is as follows: CDF(Y n ) represents a cumulative distribution function probability, MAD represents a median absolute deviation of the log pattern quantity error, Y n is a log pattern quantity value of the first log sequence, and Meadian_ represents a median of the log pattern quantity error.

2. The method of claim 1, wherein, the quantity anomaly detection on the log quantity of the real-time log sequence comprises the following steps: obtaining a historical log sequence corresponding to a time point of the real-time log sequence according to a first preset time length; inputting the historical log sequence into a log quantity anomaly detection model to perform log quantity prediction; determining a difference between the log quantity of the real-time log sequence and a log quantity prediction value, and determining that the log quantity of the real-time log sequence is abnormal if the difference is greater than a preset threshold. 3.The method of claim 1, wherein, the feature anomaly detection on the real-time log sequence comprises the following steps: determining a log mode list from the real-time log sequence based on a solidified drain3 mode extraction algorithm, performing feature extraction on the log mode list to obtain a feature list; screening the log feature list according to a pre-defined log feature white list; inputting the screened log feature list into a classification anomaly detection model to perform feature anomaly detection. 4.The method of claim 1, wherein, after obtaining the first log sequence from the log data of the second preset time length according to the first preset time length, the method further comprises the following steps: obtaining a log mode quantity of the first log sequence, and performing low-pass filtering processing on the log mode quantity; determining a log mode quantity error after low-pass filtering processing; determining a median of the log mode quantity error and an absolute deviation of the median of the log mode quantity error; calculating a cumulative distribution function probability according to the median and the absolute deviation of the median.

5. The method of claim 1, wherein, the log quantity anomaly detection model is obtained based on the following steps: obtaining a first historical log sequence according to a first preset time length; inputting the first historical log sequence into a time series model Prophet to perform training, and obtaining the log quantity anomaly detection model. 6.The method of claim 1, wherein, the classification anomaly detection model is obtained based on the following steps: obtaining a second historical log sequence according to a first preset time length; determining a log mode list from the second historical log sequence based on a solidified drain3 mode extraction algorithm, performing feature extraction on the log mode list to obtain a feature list; screening the log feature list according to a pre-defined log feature white list; labeling the screened log features; inputting the labeled log features into a support vector machine (SVM) algorithm to perform training, and obtaining the classification anomaly detection model. The solidified drain 3 mode extraction algorithm is obtained by training a drain 3 mode extraction algorithm based on the second historical log sequence.

7. An abnormality log determination apparatus characterized by comprising: Comprise: The data acquisition module is used for acquiring real-time log sequences in a first preset time length. The abnormal quantity detection module is used for inputting the real-time log sequence into a log quantity abnormality detection model, and performing quantity abnormality detection on the log quantity of the real-time log sequence. The classification abnormality detection module is used for acquiring a first log sequence from log data in a second preset time length in a first preset time length in the case that the log quantity of the real-time log sequence is abnormal; and determining that the first log sequence is an abnormal log according to a Cauchy distribution. The time sequence abnormality detection module is used for inputting log data in a second preset time length before a current time into a time sequence abnormality detection model to perform log mode quantity distribution judgment, and determining an abnormal log in the case that the feature abnormality detection result of the real-time log sequence is abnormal. The calculation formula of the cumulative distribution function probability is: CDF(Y n ) represents a cumulative distribution function probability, MAD represents a median absolute deviation of the log pattern quantity error, Y n is a log pattern quantity value of the first log sequence, and Meadian_ represents a median of the log pattern quantity error.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the abnormal log determination method in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the abnormal log determination method in any one of claims 1 to 6.

Images