A method, device, electronic device and storage medium for detecting log data

Abstract

The embodiment of the present invention provides a method, device, electronic device and storage medium for detecting log data, the method comprising: first extracting target feature data from the log data to be detected, the target feature data including target time feature data and target behavior feature data; then inputting the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data. Through the embodiment of the present invention, it is possible to effectively detect multiple related attacks suffered by the target device within a preset time period, thereby ensuring the security of the target device.

Description

Technical Field

[0001] The present invention relates to the technical field of data detection, and in particular to a method, device, electronic device and storage medium for detecting log data. Background Art

[0002] With the continuous development of information technology and the continuous expansion of network scale, security issues have become increasingly prominent, network attacks have become more and more complex and diversified, and various viruses, loopholes, and attacks have emerged in an endless stream. Network security issues have not only seriously affected people's lives, but also posed a serious threat to social security and economic development.

[0003] Among existing security incidents, there are some premeditated, long-term malicious attacks. When analyzing a certain behavior data alone, it may be no different from normal behavior, which makes it impossible to effectively detect these long-term and hidden abnormal behaviors. Summary of the invention

[0004] In view of the above problems, a method, device, electronic device and storage medium for detecting log data are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for detecting log data, the method comprising:

[0006] Extracting target feature data from the log data to be detected, wherein the target feature data includes target time feature data and target behavior feature data;

[0007] The target feature data is input into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data.

[0008] Optionally, extracting target feature data from the log data to be detected includes:

[0009] Preprocessing the log data to be detected; the preprocessing includes at least one of the following: data cleaning and normalization;

[0010] Extract target feature data from the preprocessed log data to be detected.

[0011] Optionally, extracting target feature data from the preprocessed log data to be detected includes:

[0012] Extracting at least one feature data from the preprocessed log data;

[0013] Determine the weight of each feature data separately;

[0014] According to the weight, target feature data is determined from the at least one feature data.

[0015] Optionally, the preset log anomaly detection model is provided with a normal behavior rule library and an abnormal behavior rule library, and the method further includes:

[0016] Acquire training log data, and preprocess the training log data;

[0017] Extracting at least one training feature data from the preprocessed training log data;

[0018] Clustering the at least one training feature data to obtain a normal feature data class and an abnormal feature data class;

[0019] The normal behavior rule library is generated according to the normal feature data; and the abnormal behavior rule library is generated according to the abnormal feature data.

[0020] Optionally, the inputting the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected includes:

[0021] Determine a first distance between the target feature data and the normal behavior rule library, and a second distance between the target feature data and the abnormal behavior rule library;

[0022] When the first distance is less than a first distance threshold, and the second distance is less than a second distance threshold, generating an anomaly detection result of an unknown abnormal behavior;

[0023] When the first distance is greater than the second distance, generating an anomaly detection result of the known abnormal behavior;

[0024] When the first distance is smaller than the second distance, and the second distance is larger than a second distance threshold, an abnormality detection result of a normal behavior is generated.

[0025] Optionally, clustering the at least one training feature data comprises:

[0026] Determine a third distance value between one training feature data and another training feature data;

[0027] When the third distance value is lower than a third distance threshold, the one training feature data and the another training feature data are clustered into one category.

[0028] Optionally, when generating the normal behavior rule base according to the normal feature data; and when generating the abnormal behavior rule base according to the abnormal feature data, support is iteratively calculated using multiple threads.

[0029] An embodiment of the present invention further provides a device for detecting log data, the device comprising:

[0030] A feature extraction module is used to extract target feature data from the log data to be detected, wherein the target feature data includes target time feature data and target behavior feature data;

[0031] The detection module is used to input the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data.

[0032] Optionally, the feature extraction module includes:

[0033] A preprocessing submodule is used to preprocess the log data to be detected; the preprocessing includes at least one of the following: data cleaning and normalization;

[0034] The target feature data extraction submodule is used to extract target feature data from the preprocessed log data to be detected.

[0035] Optionally, the target feature data extraction submodule is used to extract at least one feature data from the preprocessed log data; determine the weight of each feature data respectively; and determine the target feature data from the at least one feature data according to the weight.

[0036] Optionally, the preset log anomaly detection model is provided with a normal behavior rule library and an abnormal behavior rule library, and the device further includes:

[0037] A training module is used to obtain training log data and preprocess the training log data; extract at least one training feature data from the preprocessed training log data; cluster the at least one training feature data to obtain normal feature data classes and abnormal feature data classes; generate the normal behavior rule library based on the normal feature data; and generate the abnormal behavior rule library based on the abnormal feature data.

[0038] Optionally, the detection module includes:

[0039] A distance calculation submodule, used to determine a first distance between the target feature data and the normal behavior rule library, and a second distance between the target feature data and the abnormal behavior rule library;

[0040] a first abnormality detection result generating submodule, configured to generate an abnormality detection result of an unknown abnormal behavior when the first distance is less than a first distance threshold and the second distance is less than a second distance threshold;

[0041] A second anomaly detection result generating submodule, configured to generate an anomaly detection result of a known abnormal behavior when the first distance is greater than the second distance;

[0042] The third abnormality detection result generating submodule is used to generate an abnormality detection result of normal behavior when the first distance is smaller than the second distance and the second distance is larger than a second distance threshold.

[0043] Optionally, the training module includes:

[0044] The clustering submodule is used to determine a third distance value between a training feature data and another training feature data; when the third distance value is lower than a third distance threshold, cluster the training feature data and the another training feature data into one category.

[0045] Optionally, when generating the normal behavior rule base according to the normal feature data; and when generating the abnormal behavior rule base according to the abnormal feature data, support is iteratively calculated using multiple threads.

[0046] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above log data detection method when executed by the processor.

[0047] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned log data detection method is implemented.

[0048] The embodiments of the present invention have the following advantages:

[0049] In an embodiment of the present invention, target feature data can be first extracted from the log data to be detected, and the target feature data includes target time feature data and target behavior feature data; then the target feature data is input into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data. Through the embodiment of the present invention, it is possible to effectively detect multiple related attacks suffered by the target device within a preset time period, thereby ensuring the safety of the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0051] Figure 1 is a flow chart of the steps of a method for detecting log data according to an embodiment of the present invention;

[0052] Figure 2 is a flowchart of another method for detecting log data according to an embodiment of the present invention;

[0053] Figure 3 It is a flow chart of log data detection according to an embodiment of the present invention;

[0054] Figure 4 It is a structural diagram of a log anomaly detection system according to an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of an application scenario of log anomaly detection according to an embodiment of the present invention;

[0056] Figure 6 It is a structural block diagram of a log data detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] During the operation of the target device, log data for the operation status can be generated and saved in the storage device of the target device; when it is necessary to analyze whether the target device is under attack, the log data can be obtained from the storage device, and based on the analysis of the log data, it can be determined whether the target device is under attack.

[0059] When the target device suffers from multiple related attacks within a preset time period, multiple corresponding log data may be continuously recorded; if one of the log data is directly analyzed, it may be found that this log data is normal, and then it is determined that there is no abnormality in the target device; wherein, the preset time period can be set according to actual conditions, for example: one day, one week, one month, and the embodiment of the present invention does not limit this.

[0060] However, the impact of this attack on the target device is delayed and takes some time to appear; if it is not discovered in time, the target device may not work properly after a period of time.

[0061] In order to detect such attacks, the present invention can be implemented by pre-training the log anomaly detection model using feature data including time features, so that the log anomaly detection model training can detect such attacks suffered by the target device.

[0062] Furthermore, when analyzing the log data of the target device to be detected, feature data including time features can be extracted from the log data, and then input into the pre-trained log anomaly detection model for identification. Since the pre-trained log anomaly detection model is trained based on feature data including time features, it can identify the log data of the target device in the dimension of time, thereby determining whether the target device is suffering from such an attack.

[0063] Reference Figure 1 , shows a flow chart of a method for detecting log data according to an embodiment of the present invention, which may include the following steps:

[0064] Step 101: extract target feature data from the log data to be detected, where the target feature data includes target time feature data and target behavior feature data.

[0065] Among them, the log data to be detected may refer to the log data obtained from the target device that needs to be detected, which may include at least one log data recorded by the target device within a period of time; the "period of time" can be set according to actual conditions, for example: one month, one week, etc., and the embodiment of the present invention does not limit this.

[0066] When it is necessary to detect whether a target device has been subjected to multiple related attacks within a preset period of time, log data recorded by the target device within a period of time can be obtained and used as the log data to be detected.

[0067] Then, target feature data including target time feature data and target behavior feature data may be extracted from the log data to be detected in terms of time dimension and behavior dimension.

[0068] The target time feature data may refer to feature data obtained by extracting features from the log data to be detected in the time dimension, and may refer to the time when the behavior corresponding to the target behavior feature data occurs.

[0069] The target behavior feature data may refer to feature data obtained by extracting features from the log data to be detected from the behavior dimension, which may refer to the behavior recorded in the log data to be detected; the target behavior feature data may be represented in a sequence, which is not limited in this embodiment of the present invention.

[0070] As an example: the target feature data can be recorded as M(T, K); wherein T can represent the target time feature data, and K can represent the target behavior feature data.

[0071] Step 102: input the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data.

[0072] The feature data for training may refer to feature data collected in advance, including time feature data and behavior feature data; the time feature data may refer to the time when the behavior corresponding to the behavior feature data occurs.

[0073] In practical applications, the model can be trained in advance using training feature data to obtain a preset log anomaly detection model that can identify multiple related attacks within a preset time period based on time feature data; for example, behaviors that occur more than a preset number of times within a preset time period can be regarded as multiple related attacks within a preset time period; attacks that occur more than a preset number of times within a preset time period and have low harm can also be regarded as multiple related attacks within the preset time period, and the present invention does not impose any restrictions on this.

[0074] Therefore, after obtaining the target feature data including the target time feature data, the target feature data can be input into the preset log anomaly detection model; the preset log anomaly detection model can identify the log data to be detected based on the target feature data including the target time feature data and the target behavior feature data, and output the anomaly detection result for the log data to be detected; wherein the anomaly detection result may include the anomaly detection result of abnormal behavior and the anomaly detection result of normal behavior.

[0075] When the preset log anomaly detection model determines that there is an anomaly in the log data to be detected based on the target time feature data and the target behavior feature data, an anomaly detection result of abnormal behavior can be generated; when the preset log anomaly detection model determines that the log data to be detected is normal based on the target time feature data and the target behavior feature data, an anomaly detection result of normal behavior can be generated.

[0076] In an embodiment of the present invention, target feature data can be first extracted from the log data to be detected, and the target feature data includes target time feature data and target behavior feature data; then the target feature data is input into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data. Through the embodiment of the present invention, it is possible to effectively detect multiple related attacks suffered by the target device within a preset time period, thereby ensuring the safety of the target device.

[0077] Reference Figure 2 , shows a flowchart of another method for detecting log data according to an embodiment of the present invention, which may include the following steps:

[0078] Step 201: A normal behavior rule library and an abnormal behavior rule library may be provided in the preset log anomaly detection model; training log data is obtained, and the training log data is preprocessed.

[0079] The normal behavior rule base may record association rules corresponding to normal behaviors, and the abnormal behavior rule base may record association rules corresponding to abnormal behaviors.

[0080] In practical applications, log data generated by different devices in the same time period can be obtained first.

[0081] As an example, different devices may have different manufacturers, targeted businesses, monitoring contents and scopes, which may result in different log data structures. Therefore, after obtaining the training log data, the embodiments of the present invention may first pre-process the training log data to process the log data into unified, complete and formatted events.

[0082] Specifically, the training log data can be preprocessed by data cleaning, such as filling in missing values, deleting redundant log records, smoothing noisy data, identifying or deleting outliers, and resolving inconsistencies. The training log data can also be preprocessed by normalization to convert the absolute value relationship in the data into a relative value relationship, thereby simplifying the calculation and reducing the value.

[0083] Step 202: extract at least one training feature data from the preprocessed training log data.

[0084] Then, feature extraction may be performed on the preprocessed training log data to obtain at least one training feature data; the training feature data may include corresponding time feature data and corresponding behavior feature data.

[0085] Step 203: cluster at least one training feature data to obtain a normal feature data class and an abnormal feature data class.

[0086] After obtaining at least one training feature data, the at least one training feature data may be clustered to aggregate feature data with high similarity, thereby obtaining normal feature data types and abnormal feature data types.

[0087] In one embodiment of the present invention, the training feature data may be clustered by the following sub-steps:

[0088] Sub-step 11: determining a third distance between one training feature data and another training feature data.

[0089] Specifically, one training feature data and another training feature data other than the one training feature data may be selected from the at least one training feature data, and then a third distance between the one training feature data and the other training feature data is calculated; the third distance may be measured by using the Euclidean distance.

[0090] Sub-step 12: when the third distance value is lower than the third distance threshold, clustering one training feature data and another training feature data into one category.

[0091] Then, all the training feature data may be clustered according to the third distance, for example, a K-nearest neighbor algorithm may be used for cluster analysis.

[0092] Specifically, if the third distance is lower than a preset third distance threshold, the one training feature data and the other training feature data are grouped into one category; wherein the third distance threshold can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0093] If the third distance is not lower than a preset third distance threshold, the one training feature data and the other training feature data are respectively regarded as one category.

[0094] As an example, when performing clustering, at least one training feature data can be first sorted according to time to form a training original log sequence; then, the training original log sequence can be segmented according to preset time intervals, and sorted according to the weights of the training feature data therein, so as to obtain multiple groups of training log sequences; then, the third distances between different training log sequences can be calculated respectively.

[0095] For example:

[0096] 1) Treat each training log sequence as a class and calculate the distance between two classes.

[0097] 2) Calculate the maximum distance between all elements (i.e., training feature data) between the two categories, and use the third distance threshold L as the clustering condition.

[0098] Specifically, two training log sequences M i and M j The third distance D(M i , M j ) can be calculated by the following formula:

[0099]

[0100] Among them, M i K m Represents the mth training feature data in the i-th training log sequence.

[0101] In the above formula, represents the mth event in the i-th log sequence.

[0102] 3) If D(M i , M j ) is less than L, then M i and M j Grouped into one category.

[0103] 4) After multiple clustering, given a certain training log sequence, calculate the distance between it and the representative vector of the existing class. If the distance to the nearest class is less than L, the feature vector of the training log sequence is divided into the corresponding nearest class, otherwise it is a new class.

[0104] After all the training log sequences are clustered, the distances between all the classes and the prior knowledge can be determined, and based on the distances to the prior knowledge, it can be determined whether each class is normal feature data or abnormal feature data.

[0105] Step 204: Generate a normal behavior rule base based on the normal feature data; and generate an abnormal behavior rule base based on the abnormal feature data.

[0106] Then, a corresponding normal behavior rule library may be generated according to the normal feature data, and a corresponding abnormal behavior rule library may be generated according to the abnormal feature data.

[0107] For example, the Apriori algorithm can be used to establish a normal behavior rule base and an abnormal behavior rule base; specifically:

[0108] 1) Take normal feature data or abnormal feature data as input data set X = (x 11 , x 12…… x nk ), set the indicative rules (item sets) of k variables, and get the following matrix form:

[0109]

[0110] Among them, x ij It is the indicative value of the i-th data point with respect to the j-th variable (i=1, 2...n; j=1, 2...k), represented by 0 and 1; 0 means that the condition of this variable is not met, and 1 means that the condition of this scalar is met.

[0111] 2) Based on the above item set situation, generate candidate item sets, and set the minimum support as a according to the actual situation; scan all items in all item sets, count them separately, and form a list as shown in Table 1; where support refers to the number of times the rule is verified in the data set.

[0112] Table 1:

[0113] item Item 1 …… Items How many data points include this item <![CDATA[n1]]> …… <![CDATA[n s ]]> Support <![CDATA[n1 / n]]> …… <![CDATA[n s / n]]>

[0114] The minimum support degree may be set according to actual conditions, and the embodiment of the present invention does not impose any limitation on this.

[0115] 3) According to the minimum support a, filter and merge the filtered items to obtain the candidate item set. The candidate item set must satisfy "the support of any proper subset ≥ the minimum support".

[0116] 4) Keep the minimum support unchanged and repeat the operation in step 3) above, and continue merging in this way until no further merging is possible (that is, all elements in the merged item set cannot meet the generation condition "the support of any proper subset ≥ the minimum support").

[0117] 5) Output the final item set obtained in the previous step 4) and use it as a frequent item set.

[0118] 6) Mining association rules from frequent item sets, specifically:

[0119] i. For each frequent item set f, generate all its non-empty subsets;

[0120] ii. For each non-empty subset a of f, judge according to the rules and output the association rule "a→(fa)".

[0121] 7) Repeat the previous step 6) until all required association rules are mined and the final result is output to establish a normal behavior rule base or an abnormal behavior rule base.

[0122] In practical applications, the traditional Aprior algorithm is an iterative process of mining frequent patterns. During the mining process, candidate frequent sets are continuously generated, and then the support of the candidate sets is calculated. Frequent sets are generated from the candidate sets, and then new candidate sets are generated from the frequent sets through connection and pruning steps. This is repeated until no new frequent sets can be generated and the algorithm terminates. The Apriori algorithm is a classic association rule algorithm:

[0123] The first step is to find all frequent item sets in the database through retrieval.

[0124] The second step is to generate the desired strong association rules based on the frequent item sets. The association rules require two important indicators: support and confidence. Support refers to the probability that (X, Y) appear simultaneously in all transactions, as shown in the following formula (1). This indicator eliminates meaningless item sets with low frequency by setting the minimum support threshold min sup; confidence refers to the probability that the posterior item Y occurs when the prior condition X has occurred. That is, in the transaction containing X, there is also the probability of containing Y, as shown in the following formula (2). By setting the minimum confidence threshold min con, we can further filter out the rules that do not meet the requirements.

[0125]

[0126]

[0127] N can refer to all transactions; confidence measures the accuracy of the rule, that is, the proportion of all rules that meet the given condition (i.e., the premise expressed by the "if" statement of the rule) that agree with the conclusion of the current rule. The calculation method is to first count the number of occurrences of the current rule, and then divide it by the number of rules with the same condition ("if" statement).

[0128] The traditional Apriori algorithm needs to traverse the transaction database every time it calculates the support of candidate frequent sets. It has the disadvantages of frequent database scanning, large number of candidate sets, and long time consumption. When the frequent pattern length is short and the data set is sparse and small, the performance of the Apriori algorithm is good. However, for long patterns, dense or massive data sets, the time efficiency and spatial scalability of the Apriori algorithm are challenged. In order to improve the effectiveness of the Apriori algorithm, many optimization methods of the Apriori algorithm have been proposed in the prior art to address the defects of Apriori. They mainly focus on improving the efficiency of the algorithm by controlling the size of the candidate set and reducing the number of data scans, but they still cannot achieve data mining of long-term behaviors.

[0129] Therefore, in one embodiment of the present invention, when generating a normal behavior rule base based on normal feature data, and generating an abnormal behavior rule base based on abnormal feature data, the support degree can be iteratively calculated using multiple threads.

[0130] Specifically, each time the support is iterated, multiple threads can be started for concurrent calculation, so that the performance of the hardware device can be fully utilized, thereby improving the computing efficiency.

[0131] Step 205: pre-process the log data to be detected.

[0132] Among them, the preprocessing may include at least one of the following: data cleaning and normalization.

[0133] After obtaining the log data to be tested, the log data to be tested can be preprocessed by data cleaning to discover and correct identifiable errors in the data; at the same time, the consistency of the data can be checked, and invalid values ​​and missing values ​​can be processed.

[0134] For example: filling missing values, removing redundant data, smoothing noisy data, identifying or removing outliers, and resolving inconsistencies.

[0135] Of course, the data can also be pre-processed for normalization to convert the relationship between absolute values ​​in the data into a relationship between relative values, thereby simplifying the calculation and reducing the value.

[0136] Step 206: extract target feature data from the preprocessed log data to be detected.

[0137] After the log data to be detected is preprocessed, feature extraction may be performed on the preprocessed log data to be detected, so as to extract target feature data including target time feature data and target behavior feature data from the preprocessed log data to be detected.

[0138] In one embodiment of the present invention, target feature data can be extracted through sub-steps:

[0139] Sub-step 21: extract at least one feature data from the preprocessed log data.

[0140] First, feature extraction may be performed on the preprocessed log data to be detected to extract at least one feature data including time feature data and behavior feature data.

[0141] As an example, at least one feature data may be sorted according to time to form an original log sequence.

[0142] Then, the original log sequence may be divided into multiple groups of log sequences according to a preset time interval; wherein the preset time interval may be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0143] Sub-step 22: determine the weight of each feature data respectively.

[0144] After extracting at least one feature data, the weight of each feature data can be determined respectively; specifically, TF-IDF (Term Frequency–Inverse Document Frequency) can be used to calculate the weight; the weight can be used to characterize the importance of each feature data, and the larger the weight, the higher the importance of the corresponding feature data.

[0145] Sub-step 23: Determine target feature data from at least one feature data according to the weight.

[0146] After obtaining the weight of each feature data, target feature data may be determined from at least one feature data according to the weight; the target feature data may include multiple ones.

[0147] As an example, multiple groups of log sequences may be sorted into at least one group of target log sequences according to weights; the target log sequence may include at least one target feature data.

[0148] For example: the target log sequence is M1(T1, K1), M2(T2, K2), M3(T3, K3)...M n (T n , K n );wherein, T1......Tn Can represent target time characteristic data, K1...K n Can represent target behavior characteristic data.

[0149] Step 207: Input the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected.

[0150] After obtaining the target feature data including the target time feature data, the target feature data can be input into a preset log anomaly detection model; the preset log anomaly detection model can identify the log data to be detected based on the target feature data including the target time feature data and the target behavior feature data, and output the anomaly detection result for the log data to be detected.

[0151] As an example, the target log sequence can be input into a preset log anomaly detection model; the preset log anomaly detection model can identify the log data to be detected based on the target feature data in the target log sequence, including target time feature data and target behavior feature data, and output anomaly detection results for the log data to be detected.

[0152] Based on the above preset log anomaly detection model, step 207 may include the following sub-steps:

[0153] Sub-step 31: determining a first distance between the target feature data and a normal behavior rule base, and a second distance between the target feature data and an abnormal behavior rule base.

[0154] First, a first distance G1 between the target feature data and the normal behavior rule library may be calculated, and a second distance G2 between the target feature data and the abnormal behavior rule library may be calculated.

[0155] The calculation method of the first distance G1 and the second distance G2 is similar to the calculation method of the third distance mentioned above, and will not be repeated here.

[0156] Sub-step 32: when the first distance is less than the first distance threshold and the second distance is less than the second distance threshold, generate an abnormality detection result of the unknown abnormal behavior.

[0157] Then, the first distance G1 and the first distance threshold G can be determined. Z The relationship between the second distance G2 and the second distance threshold G Y , and the relationship between the first distance G1 and the second distance G2; wherein the first distance threshold G Z and the second distance threshold G Y It can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0158] If the first distance G1 is less than the first distance threshold GZ , and the second distance G2 is less than the second distance threshold G Y , which means that the behavior corresponding to the target feature data may be either normal behavior or abnormal behavior. In order to avoid judging abnormal behavior as normal behavior and causing damage to the target device, anomaly detection results of unknown abnormal behavior can be generated in this case to remind the administrator to check the target device.

[0159] Sub-step 33: When the first distance is greater than the second distance, generate an abnormality detection result of the known abnormal behavior.

[0160] If the first distance G1 is greater than the second distance G2, it may indicate that the behavior corresponding to the target feature data is more likely to be abnormal behavior than normal behavior; at this time, an anomaly detection result of a known abnormal behavior may be generated to remind the administrator to check the target device.

[0161] Sub-step 34: when the first distance is smaller than the second distance, and the second distance is larger than a second distance threshold, generating an abnormality detection result of normal behavior.

[0162] If the first distance G1 is smaller than the second distance G2, it can indicate that the behavior corresponding to the target feature data is more likely to be a normal behavior than an abnormal behavior. At this time, it can be further determined whether the second distance G2 is smaller than the second distance threshold G Y The relationship between.

[0163] If the second distance G2 is greater than the second distance threshold G Y If , it can be said that the behavior corresponding to the target feature data is less likely to be abnormal behavior; at this time, an abnormal detection result of normal behavior can be generated.

[0164] like Figure 3 , is a flow chart of a log data detection process according to an embodiment of the present invention:

[0165] S1: Obtain log data and preprocess it;

[0166] You can first obtain the log data generated by each device over a period of time and use it as training log data;

[0167] After obtaining the training log data, you can first preprocess the training log data; for example: data cleaning, normalization, etc.

[0168] S2: Extract data features;

[0169] Then, feature extraction may be performed on the preprocessed training log data to extract at least one training feature data.

[0170] S3: Cluster analysis of data;

[0171] Then, at least one training feature data may be clustered, and training feature data with high similarity may be grouped together to divide the data into a normal feature data class and an abnormal feature data class.

[0172] S4: Build the model;

[0173] Based on the normal feature data class and the abnormal feature data class, the Apriori algorithm can be used to establish a normal behavior rule library and an abnormal behavior rule library to establish a preset log anomaly detection model.

[0174] S5: abnormal data judgment;

[0175] Afterwards, the log data to be detected can be input into a preset log anomaly detection model; the preset log anomaly detection model can calculate the target feature data obtained from the log data to be detected, including the first distance and the second distance between the normal behavior rule library and the abnormal behavior rule library, and generate anomaly detection results based on the distance and the relationship between the distance and the first distance threshold and the second distance threshold.

[0176] like Figure 4 , which is a structural diagram of a log anomaly detection system according to an embodiment of the present invention; it may include a log data acquisition and preprocessing module, a data feature extraction module, a clustering analysis module, a log anomaly detection model building module and an abnormal data judgment module.

[0177] Among them, the log data acquisition and preprocessing module can be used to obtain the log data generated by each device within a period of time and use it as training log data; after obtaining the training log data, the training log data can be preprocessed first; for example: data cleaning, normalization, etc.

[0178] The data feature extraction module can be used to perform feature extraction on the preprocessed training log data to extract at least one training feature data.

[0179] The cluster analysis module can be used to cluster at least one training feature data, and aggregate the training feature data with high similarity to divide them into normal feature data classes and abnormal feature data classes.

[0180] The log anomaly detection model building module can be used to establish a normal behavior rule library and an abnormal behavior rule library based on a normal feature data class and an abnormal feature data class, and can use the Apriori algorithm to establish a preset log anomaly detection model.

[0181] The abnormal data judgment module can be used to input the log data to be detected into a preset log anomaly detection model; the preset log anomaly detection model can calculate the target feature data obtained from the log data to be detected, including the first distance and the second distance between the normal behavior rule library and the abnormal behavior rule library, and generate anomaly detection results based on the distance and the relationship between the distance and the first distance threshold and the second distance threshold.

[0182] like Figure 5 The log anomaly detection system can obtain the log data to be detected from the target device to detect whether the target device has suffered multiple related attacks within a preset period of time; after the log anomaly detection system generates an anomaly detection result for the log data to be detected, the anomaly detection result can be sent to the administrator device to remind the administrator to check the target device.

[0183] In the embodiment of the present invention, the log data to be detected can be preprocessed first; then the target feature data can be extracted from the preprocessed log data to be detected; and then the target feature data can be input into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected. Through the embodiment of the present invention, it is possible to effectively detect multiple related attacks suffered by the target device within a preset time period, thereby ensuring the security of the target device.

[0184] When clustering the training data, clustering is performed directly based on the distance between the existing data without relying on prior knowledge, thereby reducing the dependence of clustering on prior knowledge and improving the model's detection efficiency and accuracy for unknown data.

[0185] When generating anomaly detection results, judgment is made based on the distance relationship with the normal rule base, the distance relationship with the abnormal rule base, and the relationship with the distance threshold, so that both known abnormal behaviors and unknown abnormal behaviors can be detected.

[0186] In addition, during each iteration of support calculation, multiple threads are started for concurrent calculations, so that the performance of the hardware devices can be fully utilized, thereby improving computing efficiency.

[0187] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0188] Reference Figure 6, shows a schematic diagram of the structure of a log data detection device according to an embodiment of the present invention, which may include the following modules:

[0189] The feature extraction module 601 is used to extract target feature data from the log data to be detected, where the target feature data includes target time feature data and target behavior feature data;

[0190] The detection module 602 is used to input the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period. The preset log anomaly detection model is trained based on the training feature data including time feature data and behavior feature data.

[0191] In an optional embodiment of the present invention, the feature extraction module 601 includes:

[0192] The preprocessing submodule is used to preprocess the log data to be detected; the preprocessing includes at least one of the following: data cleaning and normalization;

[0193] The target feature data extraction submodule is used to extract target feature data from the preprocessed log data to be detected.

[0194] In an optional embodiment of the present invention, the target feature data extraction submodule is used to extract at least one feature data from the preprocessed log data; determine the weight of each feature data respectively; and determine the target feature data from the at least one feature data based on the weight.

[0195] In an optional embodiment of the present invention, a normal behavior rule library and an abnormal behavior rule library are provided in the preset log anomaly detection model, and the device further includes:

[0196] The training module is used to obtain training log data and preprocess the training log data; extract at least one training feature data from the preprocessed training log data; cluster the at least one training feature data to obtain a normal feature data class and an abnormal feature data class; generate a normal behavior rule library based on the normal feature data; and generate an abnormal behavior rule library based on the abnormal feature data.

[0197] In an optional embodiment of the present invention, the detection module 602 includes:

[0198] A distance calculation submodule, used to determine a first distance between the target feature data and a normal behavior rule library, and a second distance between the target feature data and an abnormal behavior rule library;

[0199] A first abnormality detection result generating submodule, used for generating an abnormality detection result of unknown abnormal behavior when the first distance is less than a first distance threshold and the second distance is less than a second distance threshold;

[0200] A second anomaly detection result generating submodule, configured to generate an anomaly detection result of a known abnormal behavior when the first distance is greater than the second distance;

[0201] The third abnormality detection result generating submodule is used to generate an abnormality detection result of normal behavior when the first distance is smaller than the second distance and the second distance is larger than the second distance threshold.

[0202] In an optional embodiment of the present invention, the training module includes:

[0203] The clustering submodule is used to determine a third distance value between one training feature data and another training feature data; when the third distance value is lower than a third distance threshold, clustering the one training feature data and the other training feature data into one category.

[0204] In an optional embodiment of the present invention, the training module is used to iteratively calculate the support using multiple threads when generating a normal behavior rule base based on normal feature data; and generating an abnormal behavior rule base based on abnormal feature data.

[0205] In an embodiment of the present invention, target feature data can be first extracted from the log data to be detected, and the target feature data includes target time feature data and target behavior feature data; then the target feature data is input into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data. Through the embodiment of the present invention, it is possible to effectively detect multiple related attacks suffered by the target device within a preset time period, thereby ensuring the safety of the target device.

[0206] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above log data detection method when executed by the processor.

[0207] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned log data detection method is implemented.

[0208] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0209] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0210] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0211] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0212] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0213] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0214] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0215] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0216] The above provides a detailed introduction to a log data detection method, device, electronic device and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting log data, characterized in that: The method comprises: Extracting target feature data from the log data to be detected, wherein the target feature data includes target time feature data and target behavior feature data; Inputting the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data; Wherein, the preset log anomaly detection model is provided with a normal behavior rule library and an abnormal behavior rule library, and the method further comprises: Acquire training log data, and preprocess the training log data; Extracting at least one training feature data from the preprocessed training log data; Clustering the at least one training feature data to obtain a normal feature data class and an abnormal feature data class; Generating the normal behavior rule base according to the normal feature data; and generating the abnormal behavior rule base according to the abnormal feature data; The step of inputting the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected includes: Determine a first distance between the target feature data and the normal behavior rule library, and a second distance between the target feature data and the abnormal behavior rule library; When the first distance is less than a first distance threshold, and the second distance is less than a second distance threshold, generating an anomaly detection result of an unknown abnormal behavior; When the first distance is greater than the second distance, generating an anomaly detection result of the known abnormal behavior; When the first distance is less than the second distance, and the second distance is greater than a second distance threshold, generating an abnormality detection result of normal behavior; The step of clustering the at least one training feature data comprises: Determine a third distance value between one training feature data and another training feature data; When the third distance value is lower than a third distance threshold, clustering the one training feature data and the another training feature data into one category; The step of determining a third distance value between one training feature data and another training feature data comprises: Sorting at least one of the training feature data according to time to form a training original log sequence; The original training log sequence is segmented according to a preset time interval, and sorted according to the weights of the training feature data therein to obtain multiple groups of training log sequences; Respectively calculating the third distance values ​​between different training log sequences; The third distance value is calculated according to the following formula: Among them, D(M i , M j ) are two training log sequences M i and M j The third distance value between i K m It is the mth training feature data in the i-th training log sequence.

2. The method according to claim 1, characterized in that The step of extracting target feature data from the log data to be detected includes: Preprocessing the log data to be detected; the preprocessing includes at least one of the following: data cleaning and normalization; Extract target feature data from the preprocessed log data to be detected.

3. The method according to claim 2, characterized in that The step of extracting target feature data from the preprocessed log data to be detected includes: Extracting at least one feature data from the preprocessed log data; Determine the weight of each feature data separately; According to the weight, target feature data is determined from the at least one feature data.

4. The method according to claim 1, characterized in that When the normal behavior rule base is generated according to the normal feature data; and when the abnormal behavior rule base is generated according to the abnormal feature data, support is iteratively calculated using multiple threads.

5. A log data detection device, characterized in that: The device comprises: A feature extraction module is used to extract target feature data from the log data to be detected, wherein the target feature data includes target time feature data and target behavior feature data; A detection module, used for inputting the target feature data into a preset log anomaly detection model to obtain an anomaly detection result for the log data to be detected; the anomaly detection result is used to indicate whether the target device corresponding to the log data to be detected has suffered multiple related attacks within a preset time period, and the preset log anomaly detection model is trained based on training feature data including time feature data and behavior feature data; A training module is used to obtain training log data and preprocess the training log data; extract at least one training feature data from the preprocessed training log data; cluster the at least one training feature data to obtain a normal feature data class and an abnormal feature data class; generate a normal behavior rule library based on the normal feature data; and generate an abnormal behavior rule library based on the abnormal feature data; Wherein, the detection module includes: A distance calculation submodule, used to determine a first distance between the target feature data and a normal behavior rule library, and a second distance between the target feature data and an abnormal behavior rule library; A first abnormality detection result generating submodule, used for generating an abnormality detection result of unknown abnormal behavior when the first distance is less than a first distance threshold and the second distance is less than a second distance threshold; A second anomaly detection result generating submodule, configured to generate an anomaly detection result of a known abnormal behavior when the first distance is greater than the second distance; a third abnormality detection result generating submodule, configured to generate an abnormality detection result of a normal behavior when the first distance is less than the second distance and the second distance is greater than a second distance threshold; Wherein, the training module includes: A clustering submodule, used to determine a third distance value between one training feature data and another training feature data; when the third distance value is lower than a third distance threshold, clustering the one training feature data and the other training feature data into one category; The step of determining a third distance value between one training feature data and another training feature data comprises: Sorting at least one of the training feature data according to time to form a training original log sequence; The original training log sequence is segmented according to a preset time interval, and sorted according to the weights of the training feature data therein to obtain multiple groups of training log sequences; Respectively calculating the third distance values ​​between different training log sequences; The third distance value is calculated according to the following formula: Among them, D(M i , M j ) are two training log sequences M i and M j The third distance value between i K m It is the mth training feature data in the i-th training log sequence.

6. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the log data detection method according to any one of claims 1 to 4 when executed by the processor.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting log data according to any one of claims 1 to 4 is implemented.

Images