Network attack type identification method and system based on log analysis

Abstract

The invention relates to the technical field of network security, in particular to a network attack type identification method and system based on log analysis, and the method comprises the steps: carrying out the preprocessing of original log data, removing time information, and retaining the structural information of a log event and a log variable; acquiring attack action data from the log data with the labels; gradually cleaning data through attack action semantic analysis, global analysis, long term analysis and short term analysis, finally constructing an attack feature tree, and generating a corresponding weight for each action node in the attack feature tree; and based on the attack feature tree, performing similarity judgment on the log item sequence after the attack occurs so as to identify the attack type. According to the method, through log analysis and multi-dimensional feature processing, interference caused by network attack polymorphism is effectively reduced, and the accuracy and reliability of attack detection are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to a network attack type identification method and system based on log analysis. BACKGROUND

[0002] At present, the methods for identifying and classifying network attack actions are as follows:

[0003] (1) Based on a convolutional neural network, attack features are learned from attack sequence fragments, so as to detect attacks. (2) Attack actions are described in semantics, and then a classifier is used to classify attacks. (3) Genetic algorithms and clustering algorithms are used to detect attack actions. (4) K-Nearest Neighbor algorithm and Apriori algorithm are used to detect attack actions. (5) Artificial neural networks are used to detect DDos attacks.

[0004] The existing network attack action detection and identification methods mainly face the following problems: attack actions are polymorphic. Each network attack does not strictly follow a certain specification, so even if it belongs to the same type of network attack, each network attack action is not the same. In order to evade detection, some network attacks will use disguising, impersonation and other means to disguise the network attack as a legal behavior or other attack action. When the difference is not obvious, it may not have a significant impact on network attack action detection and identification. However, even if the difference is small, this polymorphism can still interfere with network attack action detection and identification, and even cause detection and identification errors.

[0005] Network attacks generally have polymorphic characteristics. For example, a network attack can represent a sequence of actions A = {a1, a2, …, a n}, and the diversity of network attacks can be represented as follows:

[0006] (1) Some interference actions are added to A, such as A = {a1, a2, …, b1, …, a n}, where b1 can be an interference action introduced by the attacker, or it can be an attack habit unique to others, but it is different from the standard attack action sequence.

[0007] (2) Some unnecessary actions are deleted from A, such as A = {a1, a3, …, a n}, where a2 is deleted from the standard attack action sequence, and a2 can be just some auxiliary action of the attack, and its existence does not determine whether the attack is successful or not.

[0008] (3) Some actions are deformed, such as A = {a1, a’2, …, a nHere, a'2 is the transformed action, which is similar to the original action and does not affect the attack, but interferes with the method of strictly implementing the detection according to the action token (because in the eyes of these algorithms, a'2 and a2 are two completely irrelevant actions). SUMMARY

[0009] The present application aims to solve the problem that it is difficult to effectively identify and distinguish the interference, transformation or irrelevant action in the attack action sequence when facing the polymorphism of network attacks, resulting in insufficient accuracy and reliability of attack detection, and proposes a network attack type identification method and system based on log analysis, which effectively reduces the interference caused by network attack polymorphism through log analysis and multi-dimensional feature processing, and significantly improves the accuracy and reliability of attack detection.

[0010] To achieve the above purpose, the technical scheme adopted is:

[0011] The present application provides a network attack type identification method based on log analysis, comprising the following steps:

[0012] The original log data is preprocessed to remove time information and retain the structured information of log events and log variables;

[0013] Attack action data is obtained from labeled log data; through attack action semantic analysis, global analysis, long-term analysis and short-term analysis, the data is gradually cleaned, and finally an attack feature tree is constructed, and corresponding weights are generated for each action node in the attack feature tree;

[0014] Based on the attack feature tree, the log item sequence after the attack is determined for similarity, so as to identify the attack type.

[0015] According to the network attack type identification method based on log analysis of the present application, further, the attack action data obtained from the labeled log data comprises:

[0016] The expert labels the preprocessed log, and labels the attack type corresponding to the abnormal log item;

[0017] The labeled log is traversed, and for each labeled abnormal log item, the corresponding log item sequence is obtained by backtracking according to the set backtracking length of 6;

[0018] The obtained log item sequence and the corresponding attack type are combined to form attack action data, and are stored in an attack action data set.

[0019] According to the network attack type identification method based on log analysis of the present application, further, the attack action semantic analysis comprises:

[0020] The SBERT model is used for sentence vector generation of the log items, the log items are single elements in a log sequence, each log item represents a certain action, and a corresponding sentence vector of each log item is obtained;

[0021] The similarity of the sentence vectors is calculated based on cosine similarity, and similar log items are merged.

[0022] According to the network attack type identification method based on log analysis, further, the attack action global analysis includes:

[0023] The global analyzer is constructed based on LSTM, the input parameter is a 6-dimensional vector, and the training data set is the preprocessed log;

[0024] The sequence composed of the 6 log items obtained by backtracking in the attack action data set is input into the global analyzer, and the probability of the next log item is predicted;

[0025] If the ratio of the sum of the predicted abnormal log item probabilities to the number of abnormal log items is lower than a preset threshold, all log item sequences containing the 6 log items are filtered.

[0026] According to the network attack type identification method based on log analysis, further, the attack action long-term analysis includes:

[0027] The long-term analyzer is constructed based on LSTM, the input parameter is a 5-dimensional vector, and the training data set is the attack action data set;

[0028] The log item sequence in the attack action data set is analyzed, the length of the candidate log item sequence is extracted from the sequence of 6 log items obtained by backtracking, and the long-term analyzer is input for prediction analysis;

[0029] If the ratio of the sum of the predicted abnormal log item probabilities to the number of abnormal log items is greater than a preset threshold, the candidate log item sequence is added to the attack action long-term set.

[0030] According to the network attack type identification method based on log analysis, further, the attack action short-term analysis includes:

[0031] The short-term analyzer I is constructed based on LSTM, the input parameter is a 2-dimensional vector, the training data set is the preprocessed log, and the preprocessed log is traversed, any two continuous log item sequences {t i ,t i+1} are taken, input into the short-term analyzer I, and the probability of the next log item t i+2 is predicted, if the prediction probability exceeds a threshold, it is considered that {t i ,t i+1 ,t i+2is a strong correlation short sequence, and it is added to the set D_tmp;

[0032] The short-term analyzer II takes the short sequence and the attack action long-term set as input, checks whether the short sequence is a subsequence of the attack action long-term set, and if so, removes the subsequence from the attack action long-term set, but retains the last log item of the subsequence.

[0033] According to the network attack type identification method based on log analysis, further, the attack feature tree construction includes:

[0034] Determine the attack type to be analyzed, find the corresponding attack action sequence set;

[0035] Based on the attack action sequence set, the attack action sequence is analyzed in reverse order, and the attack feature tree is gradually constructed.

[0036] According to the network attack type identification method based on log analysis, further, the attack action weight generation includes:

[0037] The TFIDF algorithm is used to calculate the attack action weight: TFIDF=tf*idf, where the TFIDF value is the attack action weight, the tf value is the in-degree of the node in the attack feature tree, and the idf value is calculated according to the frequency of the node in other attack feature trees.

[0038] According to the network attack type identification method based on log analysis, further, the attack type identification includes:

[0039] Obtain the corresponding log item sequence from the log after the attack occurs;

[0040] Branch matching is performed on the log item sequence to be identified and the attack feature tree, and branch weight and matching weight are calculated;

[0041] The matching results of multiple branches are fused by cosine similarity, and the attack type corresponding to the attack feature tree with a similarity value greater than a set threshold is output; the similarity calculation formula is as follows:

[0042]

[0043] Where, val_lp i is the i th branch weight, which is obtained by calculating the sum of the attack action weights of all nodes on the branch; val_pp i is the matching weight of the attack action sequence on the i th branch, which is obtained by calculating the sum of the attack action weights of the matching nodes.

[0044] Further, the present application also provides a log analysis-based network attack type identification system for implementing the above log analysis-based network attack type identification method, and the system comprises:

[0045] a log preprocessing module for preprocessing original log data, removing time information, and retaining structured information of log events and log variables;

[0046] an attack feature acquisition module for acquiring attack action data in labeled log data; gradually cleaning data through attack action semantic analysis, global analysis, long-term analysis, and short-term analysis, and finally constructing an attack feature tree and generating corresponding weights for each action node in the attack feature tree;

[0047] an attack type identification module for determining similarity of log item sequences after an attack based on the attack feature tree, thereby identifying the attack type.

[0048] The above technical solution has the following beneficial effects:

[0049] The polymorphism of network attacks leads to the difficulty of network attack identification. Although network attacks have certain processes and specifications, some attack libraries also describe them as necessary, but real network attacks often deviate from detection expectations, especially some network attacks that lack clear judgment criteria. The present application extracts key attack features from network attacks with polymorphism by deeply analyzing attack traces in log data, using long short-term memory network (LSTM) and semantic analysis technology, and constructing an attack feature tree. In the attack action data acquisition stage, log item sequences related to attacks are acquired through expert marking and backtracking mechanism. In the attack feature acquisition stage, multi-level filtering mechanisms such as attack action semantic analysis, global analysis, long-term analysis, and short-term analysis are used to gradually remove noise log items and retain log item sequences closely related to attacks. Finally, attack type identification is performed based on the attack feature tree, and the attack type is accurately identified through similarity determination.

[0050] This series of technical means effectively solve the problem of insufficient detection accuracy and reliability caused by attack polymorphism in the prior art. The attack feature tree retains log item sequences closely related to attacks and removes a large number of noise and interference items. This simplified attack feature tree can more accurately reflect the core features of attacks, thereby reducing the possibility of misjudgment and omission in the detection process and improving the accuracy and reliability of attack identification. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application will be briefly introduced below. The drawings are only used to show some embodiments of the present application, and the present application is not limited to the drawings.

[0052] Figure 1 is a flowchart of a network attack type identification method based on log analysis according to an embodiment of the present application;

[0053] Figure 2 is an attack feature extraction schematic diagram according to an embodiment of the present application;

[0054] Figure 3 is an attack discrimination schematic diagram according to an embodiment of the present application;

[0055] Figure 4 is an example of log original text of an Android log according to an embodiment of the present application;

[0056] Figure 5 is a flowchart of attack sample generation according to an embodiment of the present application;

[0057] Figure 6 is a log item sequence of attack x according to an embodiment of the present application;

[0058] Figure 7 is a flowchart of attack action semantic analysis according to an embodiment of the present application;

[0059] Figure 8 is an attack action global analysis schematic diagram according to an embodiment of the present application;

[0060] Figure 9 is an attack action long item analysis schematic diagram according to an embodiment of the present application;

[0061] Figure 10 is an attack action short item analysis schematic diagram according to an embodiment of the present application;

[0062] Figure 11 is an attack feature tree generation schematic diagram according to an embodiment of the present application;

[0063] Figure 12 is an attack feature tree generation example one according to an embodiment of the present application;

[0064] Figure 13 is an attack feature tree generation example two according to an embodiment of the present application;

[0065] Figure 14 is an attack type identification example diagram according to an embodiment of the present application;

[0066] Figure 15 is a branch matching score example diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following clearly and completely describes example schemes of embodiments of the present application with reference to the accompanying drawings of specific embodiments of the present application. Unless otherwise defined, technical terms or scientific terms used in the present application should be understood as their common meanings by those skilled in the art.

[0068] Network attacks have polymorphism, and attack action sequences of the same type of network attack are usually not exactly the same. In order to evade detection, attackers often use various disguising techniques, and the polymorphism of network attacks is even more serious, which poses a challenge to the implementation of attack type recognition algorithms based on attack action characteristic codes. The present application is directed to network attacks with polymorphism, and proposes a network attack type recognition method based on log analysis.

[0069] As shown in Figure 2 , the polymorphism of network attacks is that the same network attack can have multiple different attack action sequences, and these action sequences will be reflected in the log items related to the attack. The log is the data output by the software, reflecting the software state and environmental information. Network attacks often leave traces in the log, from which network attacks can be found. The polymorphism of network attacks is mapped to the log and will be manifested as irregular mixing of log items related to the attack in normal log items.

[0070] The difficulty in finding polymorphic network attacks from logs is how to find log items closely related to a specified attack from the logs. In addition to log items closely related to the attack, there are also some log items that may be related to the attack and noise log items. The polymorphism of attack behavior causes the action sequence of each attack to be different, so the corresponding log items will also be different. The degree of relevance of each log item to the attack is not completely consistent, and further analysis of the degree of relevance of the log items is needed, and an attack feature tree (each node of the attack feature tree is a log item) is constructed.

[0071] Attack discrimination is shown in Figure 3 . After the occurrence of the attack to be determined, corresponding traces will be left in the log. Similarity determination is performed between the log and the attack feature tree to which the attack feature belongs, to analyze whether the current log trace is caused by which attack. One attack can have one or more attack feature trees, and multiple attack feature trees are needed to identify multiple attacks. Since the present application is designed for polymorphic network attacks, some unknown attacks can also be discovered by the present application.

[0072] Based on the above principle, as shown in Figure 1 , the method of the embodiment of the present application specifically includes the following steps:

[0073] Step S101, pre-process the original log data, remove the time information, and retain the structured information of the log events and log variables.

[0074] The embodiment of the present application uses the existing log preprocessing method to preprocess the log. The log original text contains several information, and the importance of these information for subsequent analysis is different. Figure 4 The log original text is the log of Android, which contains the time information "03-17 16:13:39.069", and the time information is often discarded in most subsequent analysis, so it is usually filtered in preprocessing. More attention is paid to the content such as "WindowManager:orientation change is complete, call stop Freezing Display Locked". These contents record the event information completed by the software, and also contain the corresponding variable information, which is the part of the subsequent log analysis that is focused on.

[0075] The embodiment of the present application does not focus on the attack action sensitive to the time information. Since the log preprocessing removes the time information in the log original text, only the chronological relationship is retained (the log is strictly recorded in chronological order, so although the time information is not retained, the log item recorded in the front must occur before the log item recorded in the back). In view of this, the embodiment of the present application does not analyze the attack time, and the attack action related to the time is not within the scope of the present application. There are various forms of log preprocessing algorithms, and the embodiment of the present application does not make any requirements thereon. The commonly used regular expression can be used for processing. The embodiment of the present application only requires that the structured information such as log event and log variable is obtained after log preprocessing.

[0076] In step S102, attack action data is obtained in the labeled log data; through attack action semantic analysis, global analysis, long item analysis and short item analysis, the data is gradually cleaned, and finally an attack feature tree is constructed, and corresponding weights are generated for each action node in the attack feature tree.

[0077] The attack feature collection is based on the following assumptions. First, the noise and other interference factors in the log are occasional. The log noise only appears in the attack sample occasionally, and does not always appear in the sample. The concurrency of the log output also has a random feature, although the action information of the attack and other log information may be output to the log, but these information crossed with the attack action information will not always be the same. Second, we believe that the collected sample data is benign. It is called that the sample data is benign, which means that the sample data is not deliberately forged by the attacker, although there are some interference actions, but not maliciously placed by the attacker, so as to affect our analysis. That is, there is no problem of malicious sample.

[0078] As Figure 5As shown, in order to obtain the attack feature tree, seven links need to be completed. Attack action data collection collects the log item sequence related to the log in the marked log. Attack action semantic analysis combines the log items with similar semantics, reducing the interference of similar actions on attack feature collection. Attack action global analysis, attack action long item analysis, and attack action short item analysis further clean the collected data from different angles. Attack feature tree is constructed on the basis of cleaning. Attack action weight generation generates corresponding weights for each action node in the attack feature tree to identify the importance of different actions for attack recognition.

[0079] (1) Attack action data collection

[0080] The embodiment of the application obtains attack action data in labeled log data. The preprocessed log data is a continuous ordered set of multiple log items. These log data cannot be directly used to identify attack actions. The embodiment of the application requires experts to mark attacks contained in the log data, and the marked object is a certain log item (the preprocessed log contains multiple log items, each log item represents a certain action, and is a certain action reflected in the log). As shown in the figure, the expert marks a certain log item because of the abnormal event caused by attack x. Of course, different attack types (such as attack x and attack y) may cause the same abnormal event, but the traces left by these attack types in the log may be different. Figure 6

[0081] The application obtains the original attack action data according to the principle of backtracking. The attack action data is a binary tuple, including an attack type and a log item sequence, represented as (At, {t1, t2, t3, …, t6, t x}), where At is the attack type, and {t1, t2, t3, …, t6, t x} is the log item sequence. The log item sequence is obtained by backtracking according to the abnormal log item. The default backtracking length of the application is 6. It can be adjusted according to the characteristics of the attack to be identified. Generally speaking, the log item reflecting the attack in the log will not exceed 5. Considering noise and output concurrency, irrelevant log items may be inserted in the log item, so the default is set to 6.

[0082]

[0083]

[0084] (2) Attack action semantic analysis

[0085] ​In order to find similar attack actions, the attack actions are semantically analyzed, and similar attack actions are merged. Similar attack actions are mapped to log items, that is, similar log items. Attack action semantic analysis is essentially an analysis of the similarity between log items. If log item A and log item B only have slight differences, such as "Process A is running" and "Process A is opening", they can be considered similar.

[0086] As shown in Figure 7 , the flow of attack action semantic analysis is as follows: the preprocessed log contains a plurality of log items, the sentence vector generation uses the SBERT model to obtain the sentence vector corresponding to each log item. The similarity of these sentence vectors is calculated based on the cosine similarity, and similar log items are obtained. These similar log items are considered to be interference caused by attack polymorphism, so attack action data cleaning is required, and similar log items in the attack action data obtained in step (1) are merged.

[0087] The embodiment of the present application uses the SBERT model to obtain the sentence vector of the log item. After the pre-log preprocessing link, the log item only records the description of a certain event, which can be regarded as a sentence. The present application uses all preprocessed log items as training data sets, inputs according to the requirements of the SBERT model, and obtains the sentence vector reflecting the log item. The pre-trained SBERT model can be used to skip the training link and directly obtain the sentence vector. It is recommended to use the SBERT model that has been trained with a large amount of natural language. The embodiment of the present application does not require the SBERT model itself, but only uses the SBERT model.

[0088] The cosine similarity is used to determine the similarity of the log item sentence vector. Suppose the sentence vector of log item A is A={a1, a2,…,a n}, and the sentence vector of log item B is b={b1, b2,…,b n}, then the calculation formula of their cosine similarity is:

[0089]

[0090] If sim(A, B) is less than the threshold th_sim, then the two log items are considered similar and can be merged. The default th_sim is 0.8.

[0091] Attack action data cleaning is to merge similar log items in the attack action data obtained in step (1). According to the obtained similar log items, the existing similar log items are found through multiple scans, and any log item in the similar log items is used to replace the other similar log items. After cleaning, any log item in the attack action data is not similar to other log items.

[0092] In summary, the attack action semantic analysis process is given.

[0093]

[0094]

[0095] (3) Attack action global analysis

[0096] In the embodiment of the application, the attack action data is filtered by global analysis, and the log item sequence irrelevant to the attack is deleted from the attack action data set D_At. The D_At output in step (2) can contain log item sequences irrelevant to the attack. For example, the D_At contains a log item sequence {t1, t2, t3, t4, t5, t6, x}, wherein the log item x is an abnormal log item caused by a specific attack; in addition, the D_At also contains log item sequences {t1, t2, t3, t4, t5, t6, y1}, {t1, t2, t3, t4, t5, t6, y2}, {t1, t2, t3, t4, t5, t6, y3}, {t1, t2, t3, t4, t5, t6, y4} and the like; this means that {t1, t2, t3, t4, t5, t6} can not necessarily be related to x, and it is necessary to remove it.

[0097] In the embodiment of the application, the LSTM is used to construct a global analyzer to realize the attack action global analysis. The LSTM has the ability to find the relationship between each item in the time sequence, so the LSTM is used to find the attack action sequence closely related to the attack, so that the log item related to the specific attack can be found from the complex log item. When the LSTM is used, a plurality of parameters need to be set, and the embodiment of the application only requires a plurality of related parameters. The activation function of the LSTM uses the Tanh function, the initialization method uses the Xavier initialization, and the optimizer uses the stochastic gradient descent. The input parameter is set as a 6-dimensional vector, and other parameters can be adjusted as necessary according to the type and size of the input data and the hardware resources used. The LSTM training data source is the preprocessed log. The LSTM training process is similar to the traditional method, which will not be described here.

[0098] Figure 8 is an attack action global analysis schematic diagram. Based on the trained LSTM, the global analyzer is constructed, which will predict the next log item after receiving a 6-dimensional vector input. The prediction result often does not have only one log item, and usually there are a plurality of log items, and the probabilities of predicting these log items are different. Among these log items, there can be log items caused by the attack, and there can also be other irrelevant log items. According to the assumption, the noise should be occasional and of small probability, and the embodiment of the application filters the noise according to this feature.

[0099] The filtering algorithm is as follows: assuming that the log item sequence T passes through the global analyzer, the predicted log item case is as follows: the probability that the predicted log item belongs to an abnormal log item is p, and the types of abnormal log items are n in total. Then g_p=p*(1 / (n+1)) is calculated, if g_p is greater than th_g, then it is considered that T is closely related to the attack, and the log item sequence containing T is retained; otherwise, all log item sequences containing T are removed from D_At. The log item sequence retained in D_At is considered: the probability that the abnormal log item caused by the attack is closely related is high.

[0100] th_g is a threshold for judging the legitimacy of the log item sequence, which can be set by the user. th_g is set to 0.25 by default. Generally speaking, when there is only one abnormal log item, the prediction probability of the log item should be at least more than 0.5 to retain T; if there are two abnormal log items, the sum of the prediction probabilities of the two log items should be more than 0.75 to retain T. If the size of the training data set increases, the threshold can be appropriately reduced. Because when the size of the training data is larger, the proportion of noise log items to all log items will be smaller, so the threshold can be appropriately reduced.

[0101]

[0102] (4) Attack action long-term analysis

[0103] Attack action long-term analysis is used to further filter invalid log items (i.e. noise log items) contained in the log item sequence in D_At, and further screen the log item sequence in D_At. After the global analysis of the attack action, it is confirmed that the log item sequence {t1, t2, t3, t4, t5, t6} is related to an attack, but here there may still be some log items that are accidental noise and are not closely related to the attack. Based on long-term analysis, the log items can be further analyzed to filter the possible noise.

[0104] The embodiment of the application constructs a long-term analyzer based on LSTM. Unlike the LSTM used by the global analyzer, the input data used by the LSTM here is D_At, not the preprocessed log; in addition, the LSTM here requires the length of the input log item sequence to be 5. Before using the long-term analyzer, the LSTM is required to be trained with D_At as the training data set, and after the LSTM training is completed, the long-term analyzer is in a ready state.

[0105] The method of long-term analysis is as follows: Figure 9The length of the log item sequence in D_At is 6, and the length of the alternative log item sequence is 5. For example, the log item sequence in D_At is T = {t1, t2, t3, t4, t5, t6}, 5 log items are selected from T to construct 5 alternative log item sequences. The alternative log item sequence is input to the long item analyzer. If the probability of the abnormal log item predicted by the long item analyzer (the probability is calculated with reference to g_p = p * (1 / (n+1)) in global analysis) is greater than the set threshold th_l, it is considered that the alternative log item sequence is closely related to the attack, and it is added to the attack action long item set DS_At.

[0106] th_l is a long item legitimacy threshold, which can be set by the user according to the actual situation. The prediction result of the long item analyzer is not only one abnormal log item. The embodiment of the application adopts a ladder threshold, as shown below:

[0107] ① When the prediction result of the long item analyzer is only one abnormal log item, th_l = 0.7;

[0108] ② When the prediction result of the long item analyzer is two abnormal log items, th_l = 0.5;

[0109] ③ When the prediction result of the long item analyzer is three abnormal log items, th_l = 0.3;

[0110] ④ When the prediction result of the long item analyzer exceeds three abnormal log items, th_l = 1.

[0111] For the fourth case, it is indicated that the log item sequence does not meet the condition, and no long item analysis is performed. Generally, if the global analysis is strictly screened, this situation will not occur frequently.

[0112]

[0113] (5) Attack action short item analysis

[0114] Attack action short item analysis is further analysis of the sequence in DS_At. Unlike long item analysis, which is used to filter noise and find log item sequences closely related to attacks, short item analysis further analyzes the results of long item analysis and removes some irrelevant short sequences. For example, long item analysis finds that the log item sequence {t1, t2, t3, t4, t5} is related to an attack, and short item analysis finds that the log item sequence {t1, t2} is not closely related to the attack and may exist in all attacks, so it can be removed from the sequence. Therefore, {t3, t4, t5} is finally determined to be the log item sequence related to the attack.

[0115] The short item analyzer is divided into Short Item Analyzer I and Short Item Analyzer II. Short Item Analyzer I is responsible for outputting strongly correlated short sequences, while Short Item Analyzer II is responsible for removing short sequences from DS_At and outputting the attack action set DAt. Constructing Short Item Analyzer I is similar to constructing the global analyzer. Unlike the LSTM used in the global analyzer, the LSTM here uses log item sequences of length 2. The LSTM is trained as required, and after the LSTM training is complete, Short Item Analyzer I is ready. Traverse the preprocessed logs and take any two consecutive log item sequences {t}. i ,t i+1 The input is fed into the short item analyzer I to predict the next log item t. i+2 The probability of {t}, if the predicted probability exceeds the threshold, then {t} is considered to be true. i ,t i+1 ,t i+2} is a strongly correlated short sequence, and it is added to the set D_tmp.

[0116] The simplification method of DS_At by Short Item Analyzer II is as follows. Let the long item analysis result be the log item sequence T_1, and the short item analyzer I obtain a short sequence T_2. Short item analyzer II checks whether T_2 is a subsequence of T_1. If so, the subsequence is removed from T_1, but the last log item in T_2 is retained. For example: T_1 = {t2,t3,t4,t5,t6}; T_2 = {t2,t3,t4}. It is found that T_2 is a subsequence of T_1, so the subsequence is removed, but the last log item t4 is retained. Therefore, the simplified T_1 = {t4,t5,t6}.

[0117] Short item analysis, such as Figure 10 As shown. Short Item Analyzer I takes the preprocessed logs as input and identifies strongly correlated short sequences. In this embodiment of the invention, the length of the short sequence is limited to 3, that is, a sequence containing 3 log items. According to the prediction pattern of LSTM, this can be figuratively understood as {t1,t2}→t3, that is, if t1 and t2 occur, then t3 will also likely occur. Short Item Analyzer II then takes these short sequences as input, traverses DS_At, removes short sequences, and forms the attack action set DAt.

[0118] Based on the above, a short analysis process for attack actions is presented.

[0119]

[0120]

[0121] (6) Attack Feature Tree Generation

[0122] Based on DAt, the attack feature tree is generated. Although the attack action sequence is linear from a single attack, the attack path can be diverse, more like a tree. Of course, the more realistic situation is that a network structure can be presented. However, from the perspective of distinguishability and feasibility, it is more appropriate to record the attack features in a tree structure. Therefore, the embodiment of the application constructs an attack feature tree based on the repeatedly screened log items (i.e., DAt) as a benchmark for identifying attacks.

[0123] The process of generating the attack feature tree is shown in Figure 11 First, according to the attack type at, the corresponding abnormal log item sequence is determined, denoted by x. It is not excluded that one attack corresponds to multiple abnormal log items, and there can be multiple attacks corresponding to one abnormal log item. Second, search for sequences in DAt with x as the end of the log sequence and corresponding to the attack type at (the log item sequence ending with x can correspond to multiple attack paths), as shown in 1→2→3→4→5→x, a→b→c→d→5→x, e→f→g→4→5→x in the figure. Finally, take these attack action sequences as input and start to construct the attack feature tree.

[0124] The process of constructing the attack feature tree from the screened attack action sequences is described as follows. The initial condition of the attack feature tree Tree_Ack is the root node x; take a sequence Seq={x1, x2, x3, x4, x5, x} from the attack action sequence, take out the node x5 in reverse order, query whether there is x5 connected with x in Tree_Ack, if yes, continue to backtrack Seq, otherwise add the node x5; continue to backtrack Seq to complete similar work until the processing of the sequence is completed; take the next sequence from the attack action sequence set and repeat the above process to finally obtain the attack feature tree.

[0125] Figure 12 The process of generating the attack feature tree by processing the attack action sequence {1→2→3→4→5→x} is shown. In the initial condition, the attack feature tree has only one node x; analyze the nodes in the attack action sequence in reverse order, first process the node 5, at this time the attack feature tree does not have the node 5, so add the node 5 and the edge 5→x in the attack feature tree; continue to analyze the nodes in reverse order, as shown in the figure, the node 3 is added to the attack feature tree and the edge 3→4 is added according to the above principle.

[0126] Figure 13The attack action sequence {a→b→c→d→5→x} is displayed to further improve the attack feature tree. The attack action sequence {a→b→c→d→5→x} is analyzed in reverse order, the node x is processed first, at this time, the attack feature tree has the node 5, so no further processing is required; then the node d is processed in reverse order, at this time, the attack feature tree does not have the node d and the corresponding edge, so the node d and the edge d→5 are added; the node c is processed in reverse order, and the node c and the edge c→d are added to the attack feature tree in a similar manner. Similarly, when all attack action sequences are processed, the attack feature tree owned by a certain attack is basically constructed.

[0127] The process of generating the corresponding attack feature tree for a certain attack is as follows.

[0128]

[0129] (7) Attack action weight generation

[0130] The attack action weight is used to identify the importance of the attack action for identifying the attack in the embodiment of the application. The attack usually contains multiple steps of actions, and the importance of different actions for identifying the attack is obviously different, and some actions may be unique to the attack and more helpful for identifying the attack. In addition, different action sequences may be used to implement the attack, and the corresponding actions have diversity, and it is necessary to treat different actions of the attack differently. The attack action weight is used to identify the importance of these actions for identifying the attack type, so as to focus on these actions when identifying the attack, so as to improve the identification rate of the attack.

[0131] The TFIDF algorithm is used to generate the attack action weight in the embodiment of the application. In essence, the attack action weight is the TFIDF value, and TFIDF=tf*idf, so the corresponding tf and idf values need to be obtained to generate the attack action weight. For a certain attack type att_t, the process of calculating the tf value is as follows: the attack feature tree Tree_DAt(att_t) corresponding to the attack is determined, the in-degree of all nodes except the root node (the in-degree is essentially the frequency of action occurrence) is counted, and the in-degree value is the tf value of the attack action corresponding to the node, which is marked as tf(e i The process of calculating the idf value is as follows: the attack feature trees representing other attack types are searched to determine whether the nodes of Tree_DAt(att_t) are included, and the idf value is calculated according to the following formula:

[0132]

[0133] Where n is the number of all attack feature trees, and df(e i ) is the number of attack feature trees containing the node e iThe number of other attack feature trees. For example, there are currently 10 attack feature trees, and 7 of the other attack feature trees contain e, so here n = 10 and df(e) = 7.

[0134]

[0135] Step S103, based on the attack feature tree, similarity determination is performed on the log item sequence after the attack, so as to identify the attack type.

[0136] The attack type identification is completed based on the similarity determination between the attack action sequence and the attack feature tree. Ideally, the attack action sequence should match a branch of the attack feature tree. However, in reality, the attack action sequence is often difficult to completely match a branch of the attack feature tree, and most of the cases are as follows: Figure 14 The different actions of the attack action sequence are located on different sub-trees of the attack feature tree, and this "chaotic" situation increases the difficulty of attack type identification.

[0137] The embodiment of the application is matched according to the principles of "branch matching" and "fusion review". Branch matching is to match the attack sequence according to the branches of the attack feature tree, and to obtain the result of the determination according to the current branch. As shown in Figure 14 , the branches b→c→d→5→x and 1→2→3→4→5→x can be matched. Fusion review is to fuse the determination results of multiple branches for comprehensive review.

[0138] The details of branch matching are as follows: after determining the branches of the attack feature tree, the sum of the attack action weights of all nodes on the branch is counted according to the branch, and the branch weight value is obtained, which is val_l for convenience. The sum of the weights of the matched nodes is counted by matching the attack sequence on the branch, and the matching weight value is obtained, which is val_p for convenience. As shown in Figure 15 , the matched branch is x→t1→s2, and the branch weight is 0.5+0.6=1.1; the branch is matched with the attack sequence, and only node t1 is matched, so the matching weight is 0.5.

[0139] The details of fusion review are as follows. If the nodes of the attack action sequence are scattered in multiple branches of the attack feature tree, the matching scores of multiple branches are fused. The cosine similarity is used in the embodiment of the application to complete this work. The process of constructing the vectors to be compared is as follows: suppose the nodes of the attack action sequence are scattered in several branches of the attack feature tree, and are represented as {pl1, pl2, …, pln}, then the branch weights corresponding to these branches can be obtained, and a vector P = {val_lp1, val_lp2, …, val_lp n n} is formed. nAccording to the dispersion of the attack action sequence on the attack feature tree, the matching weight corresponding to the branch can be obtained, and a vector Q={val_pp1, val_pp2, …, val_ppn} is formed. n The similarity calculation formula of the two vectors is as follows:

[0140]

[0141] If sim(P, Q) is greater than a threshold th_att, it is considered that the current attack action sequence is similar to the attack feature tree, and the attack type is found. The embodiment of the application defaults th_att to 0.5.

[0142]

[0143]

[0144] Corresponding to the above method, the embodiment also discloses a network attack type identification system based on log analysis, which comprises:

[0145] A log preprocessing module is configured to preprocess original log data, remove time information, and retain structured information of log events and log variables;

[0146] An attack feature acquisition module is configured to acquire attack action data in labeled log data; through attack action semantic analysis, global analysis, long-term analysis, and short-term analysis, the data is gradually cleaned, and finally an attack feature tree is constructed, and corresponding weights are generated for each action node in the attack feature tree;

[0147] An attack type identification module is configured to determine the similarity of log item sequences after an attack based on the attack feature tree, so as to identify the attack type.

[0148] Since the traces of network attacks in the log are used as data in the application, network attacks that do not leave traces in the log cannot be identified and determined. Although this limits the application range of the application, compared with the existing attack behavior description, the log is semi-structured data, which is more conducive to data analysis and attack identification, which is also the reason why the application uses the log as the analysis data set.

[0149] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1.A method for identifying a network attack type based on log analysis, characterized in that, The method comprises the following steps: Pretreatment of original log data, removing time information, retaining structured information of log events and log variables; Obtaining attack action data in labeled log data; Through attack action semantic analysis, global analysis, long-term analysis and short-term analysis, gradually cleaning the data, finally constructing an attack feature tree, and generating corresponding weights for each action node in the attack feature tree; Based on the attack feature tree, the similarity of the log item sequence after the attack is determined, so as to identify the attack type. 2.The log analysis based network attack type identification method according to claim 1, characterized in that, The attack action data in the labeled log data comprises: Experts mark the preprocessed log, mark the attack type corresponding to the abnormal log item; Traverse the marked log, and for each marked abnormal log item, backtrack according to the set backtracking length of 6 to obtain the corresponding log item sequence; Combine the obtained log item sequence with the corresponding attack type to form attack action data, and store it in the attack action data set. 3.The log analysis based network attack type identification method of claim 2, wherein, The attack action semantic analysis comprises: Using the SBERT model to generate sentence vectors of log items, the log items are single elements in the log sequence, each log item represents a certain action, and the sentence vector corresponding to each log item is obtained; Based on the cosine similarity, the similarity of the sentence vectors is calculated, and similar log items are combined. 4.The log analysis based network attack type identification method of claim 2, wherein, The attack action global analysis comprises: Based on LSTM, a global analyzer is constructed, the input parameter is a 6-dimensional vector, and the training data set is the preprocessed log; The sequence composed of 6 log items obtained by backtracking in the attack action data set is input into the global analyzer to predict the probability of the next log item; If the ratio of the sum of the predicted abnormal log item probabilities to the number of abnormal log items is lower than the preset threshold, filter all log item sequences containing the 6 log items. 5.The log analysis based network attack type identification method of claim 2, wherein, The attack action long-term analysis comprises: Based on LSTM, a long-term analyzer is constructed, the input parameter is a 5-dimensional vector, and the training data set is the attack action data set; The log item sequence in the attack action data set is analyzed, and the length of 5 candidate log item sequences is extracted from the sequence of 6 log items obtained by backtracking, and input into the long-term analyzer for prediction analysis; If the ratio of the sum of the predicted abnormal log item probabilities to the number of abnormal log items is greater than the preset threshold, the candidate log item sequence is added to the attack action long-term set. 6.The log analysis based network attack type identification method of claim 5, wherein, The attack action short-term analysis comprises: Based on LSTM, a short-term analyzer I is constructed, the input parameter is a 2-dimensional vector, and the training data set is the preprocessed log; traverse the preprocessed log, take any two continuous log item sequences {t i ,t i+1}, input to the short-term analyzer I, predict the probability of the next log item t i+2 , if the prediction probability exceeds the threshold, it is considered that {t i ,t i+1 ,t i+2} is a strong correlation short sequence, and it is added to the set D_tmp; Short-term analyzer II takes the short sequence obtained by short-term analyzer I and the attack action long-term set as input, checks whether the short sequence is a subsequence in the attack action long-term set, if yes, removes the subsequence from the attack action long-term set, but retains the last log item of the subsequence. 7.The log analysis based network attack type identification method of claim 1, wherein, The attack feature tree comprises: Determine the attack type to be analyzed, find the corresponding attack action sequence set; Based on the attack action sequence set, analyze the attack action sequence in reverse order, and gradually construct the attack feature tree. 8.The log analysis based network attack type identification method of claim 1, wherein, The attack action weight generation comprises: The attack action weight is calculated using a TFIDF algorithm: TFIDF=tf*idf, wherein the TFIDF value is the attack action weight, the tf value is the in-degree of a node in the attack feature tree, and the idf value is calculated according to the frequency of the node in other attack feature trees. 9.The log analysis based network attack type identification method of claim 8, wherein, The attack type recognition comprises: obtaining a corresponding log item sequence from logs after an attack occurs; performing branch matching on the log item sequence to be recognized and the attack feature tree, and calculating branch weight and matching weight; fusing the matching results of multiple branches through cosine similarity, and outputting an attack type corresponding to an attack feature tree with a similarity value greater than a set threshold; the similarity calculation formula is as follows: wherein val_lp i is the i-th branch weight, obtained by calculating the sum of the attack action weights of all nodes in the branch; val_pp i is the matching weight of the attack action sequence on the i-th branch, obtained by calculating the sum of the attack action weights of the matching nodes. 10.A system for log-based network attack type identification, characterized in that, The system is used for implementing the network attack type recognition method based on log analysis according to any one of claims 1-9, and comprises: a log preprocessing module, which is used for preprocessing original log data, removing time information, and retaining structured information of log events and log variables; an attack feature acquisition module, which is used for acquiring attack action data in labeled log data; through attack action semantic analysis, global analysis, long item analysis, and short item analysis, data is gradually cleaned, an attack feature tree is finally constructed, and corresponding weights are generated for each action node in the attack feature tree; an attack type recognition module, which is used for performing similarity determination on a log item sequence after an attack occurs based on the attack feature tree, so as to recognize an attack type.

Images