False alarm elimination method using graph attention auto-encoder to assist community detection
Through the graph attention autoencoder assists community detection, the characteristics and topological information of the alarm data are used to solve the problem of high IDS false alarm rate, effectively eliminate false alarms, improve work efficiency and resource allocation, and alleviate alarms fatigue.
Patent Information
- Application Number
- CN202411108808.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
AI Technical Summary
In existing network security technologies, intrusion detection systems (IDS) often generate false alarms, resulting in alarm fatigue problems, high-risk alarms may be ignored or delayed responses, and the post-processing method is based on labeling workload for large-scale network traffic datasets. Large and costly.
The graph attention autoencoder assists community detection by extracting the characteristics and topological information of the alarm data, and using the minimization reconstruction to maximize the topological loss and modularity, distinguishing between real alarms and false alarms, and realizing the elimination of alarm false alarms.
It effectively reduces the alarm false alarm rate, alleviates the alarm fatigue problem, improves work efficiency and resource allocation optimization, and reduces the workload and cost of security analysts.
Smart Images

Figure CN119011366A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security technology, and in particular to a method for eliminating false alarms by using a graph attention autoencoder to assist community detection. Background Art
[0002] With the rapid development of information technology, network security issues have become increasingly prominent. In order to meet this challenge, various network security technologies have emerged, and the Intrusion Detection System (IDS) has become one of the key means to ensure network security. When a threat is detected, the IDS will immediately issue an alarm and may take measures such as blocking the attack source and recording attack information to minimize potential security risks. However, due to the complexity and variability of network traffic and the imperfection of rule signatures, IDS often generates false alarms, forcing security analysts to spend a lot of time dealing with these false alarms, thereby reducing work efficiency. At the same time, the problem of alarm fatigue can have serious consequences for enterprise security. When analysts are overwhelmed by a large number of alarms, high-risk alarms may be ignored or delayed in response, exposing the organization to potential network threats or data leaks.
[0003] At present, there are two main methods to reduce the false alarm rate of alarms: detection technology-based and post-processing-based. The former is implemented in the detection stage, and the detection accuracy is improved by optimizing IDS configuration, thresholds and rules or using machine learning models. The latter processes the generated alarms after the detection stage, mainly using classifiers to identify false positives. Relevant research in the industry is mainly based on detection technology to alleviate the problem of alarm fatigue, but due to the combination of changes in attack strategies and human factors, methods based on detection technology will still generate a certain number of false alarms. Therefore, it is necessary to further analyze and filter alarms based on post-processing methods to reduce alarm fatigue and improve the reliability of alarms. Traditional post-processing-based methods mainly use classifiers to distinguish between real alarms and false alarms after obtaining alarms, but most methods to reduce false alarm rates are based on supervised learning. For large-scale network traffic data sets, the workload of manually labeling alarm data is very large and the cost is also high. Summary of the invention
[0004] The purpose of the present invention is to provide a method for eliminating false alarms by using a graph attention autoencoder to assist community detection. The method effectively utilizes the characteristics and topological information of the alarm data, reduces the false alarm rate, and alleviates the problem of alarm fatigue.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] A method for eliminating false alarms using a graph attention autoencoder to assist community detection, the method comprising:
[0007] Step 1: First, extract features that distinguish true alarms from false alarms from the original alarm data;
[0008] Step 2: Obtain the attribute features and topological information of the alarm data through the graph attention autoencoder, and obtain the representation of the alarm node by minimizing the reconstructed topological loss;
[0009] Step 3: Then, modularity maximization is used to assist community detection and classify true alarms and false alarms.
[0010] It can be seen from the technical solution provided by the present invention that the above method effectively utilizes the characteristics and topology information of the alarm data, reduces the false alarm rate of alarms, alleviates the problem of alarm fatigue, allows the team to focus on real security incidents, improves work efficiency and optimizes resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0012] Figure 1 A schematic flow chart of a method for eliminating false alarms using a graph attention autoencoder to assist in community detection provided by an embodiment of the present invention;
[0013] Figure 2 This is a schematic diagram of the alarm topology described in an embodiment of the present invention;
[0014] Figure 3 The figure is a schematic diagram of the encoding and decoding process described in an embodiment of the present invention. DETAILED DESCRIPTION
[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0016] like Figure 1 The figure is a flow chart of a method for eliminating false alarms using a graph attention autoencoder to assist community detection provided by an embodiment of the present invention, the method comprising:
[0017] Step 1: First, extract features that distinguish true alarms from false alarms from the original alarm data;
[0018] In this step, since the alarms generated by different network intrusion prevention systems differ in structure and content, it is necessary to convert these heterogeneous raw alarm data into a unified standard format for further analysis and processing.
[0019] First, a tuple is used to describe each raw alarm data, including timestamp, alarm category, alarm description, source IP, source port, destination IP, destination port, and protocol;
[0020] In order to distinguish the characteristics of real alarms and false alarms, five basic features are first selected, namely alarm category, alarm description, source IP, destination IP and protocol; then the contextual relationship of the alarm is captured based on the contextual features, that is, the environment, conditions and related information when the alarm occurs, to help distinguish false alarms. In the specific implementation, based on the analysis of the characteristics of real alarms and false alarms, 6 contextual features are finally extracted, and the specific contents are shown in Table 1:
[0021] Table 1. Feature description
[0022]
[0023]
[0024] Among them, for the alarm category and alarm description in the basic features, the word embedding method (Word2vec) is used to convert them into vectors; and in order to eliminate the indicator differences between different features, normalization is used to standardize the data to solve the comparability problem between data indicators.
[0025] Step 2: Obtain the attribute features and topological information of the alarm data through the graph attention autoencoder, and obtain the representation of the alarm node by minimizing the reconstructed topological loss;
[0026] In this step, a graph embedding model is introduced, which combines the graph attention network and the autoencoder to learn the attribute features and topological information of the alarm data;
[0027] Define a graph as G = {V, E, X}, where V = {v1, v2, ...v n} is the set of alarm nodes; E = {(v i , v j )|1≤i, j≤n, j≠j} is the set of edges, e ij =(v i , v j )∈E means there is a link from node v i to vj The edge; X = {x1; x2; ... x n} is the attribute matrix, x i Represents node v i The topological structure of the graph G is represented by the n×n adjacency matrix A=(a ij ) n×n It means that the definition is as follows:
[0028]
[0029] like Figure 2 The figure shows a schematic diagram of the alarm topology according to an embodiment of the present invention. Each alarm is regarded as a node in the network, and the link between the source IP and the destination IP is regarded as a topological relationship. In this way, the false alarm identification problem is transformed into a graph node classification problem. The attribute features and topological information of the alarm data are obtained by using a graph attention autoencoder (GATE). The input, including node attributes and graph structure, is reconstructed through stacked encoder / decoder layers and a self-attention mechanism.
[0030] In the encoder, the attributes of the alarm are regarded as the initial node representation, and each layer generates a new representation of the node through the neighboring node representations of the participants; in the decoder, the alarm attributes are reconstructed through the reverse encoding process; in addition, the representation of the alarm is regularized and the graph structure is reconstructed. This method can effectively obtain the attribute characteristics and topological information of the alarm data, so as to better distinguish false alarms in the network.
[0031] like Figure 3 The figure is a schematic diagram of the encoding and decoding process according to an embodiment of the present invention. The encoding and decoding process is specifically as follows:
[0032] (1) Encoder layer
[0033] A stacked two-layer encoder is used to improve learning and propagate node representations through the graph structure to produce richer node embeddings. Each layer of encoder uses the representations of adjacent nodes to generate a new representation of the node and uses an attention mechanism to determine the node v i Its neighbor node v j The correlation between ij , expressed as:
[0034]
[0035] Where W (k) , and are the trainable parameters of the k-th layer encoder; σ and They are all activation functions; is the representation of the k-th layer node;
[0036] In order to ensure that node vi The correlation coefficients of adjacent nodes are comparable, and the softmax function is used to calculate r ij Normalized, expressed as As shown below:
[0037]
[0038] Where N i Represents node v i Neighboring nodes of node v i The representation calculation process at the k-th layer encoder is as follows:
[0039]
[0040] Where W (k) is the weight matrix; σ is the activation function; after applying L layers of encoders, the output of the last layer is the final node representation;
[0041] (2) Decoder layer
[0042] To reverse the encoding process, a decoder with the same number of layers as the encoder is used to reconstruct the generated node representation. Each decoder layer reverses the process of its corresponding encoder layer, that is, each decoder layer reconstructs the representation of the node using the representation of the neighboring nodes according to the correlation of the node.
[0043] The neighboring nodes v of the k-th layer decoder j With node v i The normalized correlation coefficient is The specific calculation is as follows:
[0044]
[0045] in is the neighboring node v of the k-th layer decoder j With node v i The correlation coefficient of and are the trainable parameters of the k-th layer decoder; σ and is the activation function;
[0046] The last layer output of the encoder provides input for the decoder. According to the representation of the decoder at layer k, the representation of the node at the previous layer is obtained as As shown below:
[0047]
[0048] After applying L layers of decoders, the output of the last layer is used as the reconstructed node features; For node v iThe reconstruction attribute matrix and node attribute reconstruction loss L F The definition is as follows:
[0049]
[0050] Where X i and Represents node v i the original and reconstructed attribute matrices;
[0051] The inner product is used to calculate the cosine similarity of two vectors. This is effective when the magnitude of the distance metric of the vectors is constant. Since the latent representation already contains the topological structure and node attributes, using the inner product decoder is efficient and flexible. The loss L for topological reconstruction G As shown below:
[0052]
[0053] in is the activation function;
[0054] The loss function of the graph attention autoencoder is defined as the original adjacency matrix A and the predicted adjacency matrix A pred The reconstruction error between them is L, and the loss of topology reconstruction is G The binary cross-entropy loss function is used for calculation, and its formula is as follows:
[0055]
[0056] The topology reconstruction loss L G and node attribute reconstruction loss L F Combined to form the final reconstruction loss function L Gate for:
[0057] L Gate --L F +)'L G
[0058] Where γ is responsible for adjusting the topology reconstruction loss L G contribution.
[0059] Step 3: Then, modularity maximization is used to assist community detection and classify true alarms and false alarms.
[0060] In this step, normal network activities can be represented by strong, interconnected communities of objects. Attacks are usually composed of several events with rare properties (such as a source IP address that suddenly appears), so they are scattered in the graph, while attacks are more closely connected to each other. Community detection is equivalent to "grouping nodes". Modularity is a common indicator for measuring the quality of node grouping. The higher the modularity, the more the detected community conforms to the characteristics of "tight inside and loose outside", and the better the grouping quality.
[0061] Maximizing modularity allows nodes to be divided into different communities. In a graph with high modularity, nodes in the same community are densely connected, while nodes in different communities are sparsely connected.
[0062] Since we only need to distinguish between real alarms and false alarms, the number of divided communities is defined as 2, and the modularity Mod of the graph is defined as:
[0063]
[0064] Where B represents the modularity matrix, A is the adjacency matrix, d i Represents node v i The degree of , m represents the number of all edges in the graph; Tr(·) is the trace of the matrix; H represents the community assignment matrix, which is defined as follows:
[0065]
[0066] The value of H is 1 for node v i belongs to community k, and a value of 0 indicates that node v i Not belonging to community k;
[0067] Setting the embedding dimension to K means that the matrix Z is regarded as a community assignment matrix, i.e., H = Z, and the node embedding is obtained by optimizing the modularity;
[0068] Since the number of communities in the network is much smaller than the number of nodes, if the number of communities is used as the embedding dimension of the nodes, the learned representation is usually not enough to express rich semantic information. To solve this problem, this application also introduces a learnable fully connected layer F = R h×k Embeddings are mapped to community assignments; here, h represents the number of communities and k represents the desired higher dimensionality;
[0069] Fully connected layer F = R h×k The role of is to map the low-dimensional representation of each community to a higher-dimensional space, so that the embedding of the node can capture the structure and semantic information of the network. The loss function L that maximizes the modularity is Mod The definition is as follows:
[0070]
[0071] The goal of the final model is to minimize the following loss function:
[0072] L=L Gate -βL Mod
[0073] Where β represents the hyperparameters of different loss weights;
[0074] The distinction between real alarms and false alarms is achieved based on the results of community classification. Modularity is used to measure the difference between the internal connections of the community and the internal connections of the same division in a random graph. Real attackers usually form highly concentrated subgroups in the network, and the connections between them are frequent and strong. This close community connection can be reflected by higher network connectivity and denser interaction patterns, thereby helping to identify real attack activities, that is, distinguishing real alarms.
[0075] Through the above three stages, we can make full use of the attribute characteristics and topological structure of the alarm, and distinguish true and false alarms through node representation learning and community detection.
[0076] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.
[0077] In summary, the method described in the embodiment of the present invention solves the problem of often ignoring the topological structure between alarm data when eliminating false alarms, and fully considers the distribution differences between true alarms and false alarms; through this method, the topological relationship and attribute characteristics between the alarm data can be better utilized, the distinction between true alarms and false alarms can be improved, thereby improving the effectiveness and feasibility of eliminating false alarms.
[0078] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.
[0079] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.
Claims
1. A method for eliminating false alarms using a graph attention autoencoder to assist community detection, characterized in that: The method comprises: Step 1: First, extract features that distinguish true alarms from false alarms from the original alarm data; Step 2: Obtain the attribute features and topological information of the alarm data through the graph attention autoencoder, and obtain the representation of the alarm node by minimizing the reconstructed topological loss; Step 3: Then, modularity maximization is used to assist community detection and classify true alarms and false alarms.
2. According to claim 1, the method for eliminating false alarms using a graph attention autoencoder to assist community detection is characterized in that: In step 1, a tuple is used to describe each raw alarm data, including timestamp, alarm category, alarm description, source IP, source port, destination IP, destination port, and protocol; In order to distinguish the features of real alarms from false alarms, we first select five basic features, namely alarm category, alarm description, source IP, destination IP and protocol; then we capture the contextual relationship of the alarm based on the context features, that is, based on the environment, conditions and related information when the alarm occurs, to help distinguish false alarms; Among them, for the alarm category and alarm description in the basic features, the word embedding method is used to convert them into vectors; and in order to eliminate the indicator differences between different features, normalization is used to standardize the data to solve the comparability problem between data indicators.
3. According to claim 1, the method for eliminating false alarms using a graph attention autoencoder to assist community detection is characterized in that: In step 2, a graph embedding model is introduced, which combines the graph attention network and the autoencoder to learn the attribute features and topological information of the alarm data; Define the graph as G = {V, E, X}, where V = {v1, v2, ...v n } is the set of alarm nodes; E = {(v i ,v j )|1≤i,j≤n,i≠j} is the set of edges, e ij =(v i ,v j )∈E means there is a link from node v i to v j The edge; X = {x1; x2; ... x n } is the attribute matrix, x i Represents node v i The topological structure of the graph G is represented by the n×n adjacency matrix A=(a ij ) n×n It means that the definition is as follows: Each alarm is regarded as a node in the network, and the link between the source IP and the destination IP is regarded as a topological relationship. In this way, the false alarm identification problem is transformed into a graph node classification problem. The graph attention autoencoder is used to obtain the attribute features and topological information of the alarm data, and the input is reconstructed through stacked encoder / decoder layers and self-attention mechanism, including node attributes and graph structure. In the encoder, the attributes of the alarm are considered as the initial node representation, and each layer generates a new representation of the node through the neighboring node representations of the participants; in the decoder, the alarm attributes are reconstructed through the reverse encoding process; The encoding and decoding process is as follows: (1) Encoder layer A stacked two-layer encoder is used to improve learning and propagate node representations through the graph structure to produce richer node embeddings. Each layer of encoder uses the representations of adjacent nodes to generate a new representation of the node and uses an attention mechanism to determine the node v i Its neighbor node v j The correlation between ij , expressed as: Where W (k) , and are the trainable parameters of the k-th layer encoder; σ and They are all activation functions; is the representation of the k-th layer node; In order to ensure that node v i The correlation coefficients of adjacent nodes are comparable, and the softmax function is used to calculate r ij Normalized, expressed as As shown below: Where N i Represents node v i Neighboring nodes of node v i The representation calculation process at the k-th layer encoder is as follows: Where W (k) is the weight matrix; σ is the activation function; after applying L layers of encoders, the output of the last layer is the final node representation; (2) Decoder layer To reverse the encoding process, a decoder with the same number of layers as the encoder is used to reconstruct the generated node representation. Each decoder layer reverses the process of its corresponding encoder layer, that is, each decoder layer reconstructs the representation of the node using the representation of the neighboring nodes according to the correlation of the node. The neighboring nodes v of the k-th layer decoder j With node v i The normalized correlation coefficient is The specific calculation is as follows: in is the neighboring node v of the k-th layer decoder j With node v i The correlation coefficient of and are the trainable parameters of the k-th layer decoder; σ and is the activation function; The last layer output of the encoder provides input for the decoder. According to the representation of the decoder at layer k, the representation of the node at the previous layer is obtained as As shown below: After applying L layers of decoders, the output of the last layer is used as the reconstructed node features; For node v i The reconstruction attribute matrix and node attribute reconstruction loss L F The definition is as follows: Where X i and Represents node v i the original and reconstructed attribute matrices; The loss of topology reconstruction L G As shown below: in is the activation function; The loss function of the graph attention autoencoder is defined as the original adjacency matrix A and the predicted adjacency matrix A pred The reconstruction error between them is L, and the loss of topology reconstruction is G The binary cross entropy loss function is used for calculation, and its formula is as follows: The topology reconstruction loss L G and node attribute reconstruction loss L F Combined to form the final reconstruction loss function L Gate for: THE Gate =L F +γL G Where γ is responsible for adjusting the topology reconstruction loss L G contribution.
4. According to claim 3, the method for eliminating false alarms using a graph attention autoencoder to assist community detection is characterized in that: In step 3, modularity is maximized so that nodes are divided into different communities. In a graph with high modularity, nodes in the same community are densely connected, while nodes in different communities are sparsely connected. Since we only need to distinguish between real alarms and false alarms, the number of divided communities is defined as 2, and the modularity Mod of the graph is defined as: Where B represents the modularity matrix, A is the adjacency matrix, d i Represents node v i The degree of , m represents the number of all edges in the graph; Tr(·) is the trace of the matrix; H represents the community assignment matrix, which is defined as follows: The value of H is 1 for node v i belongs to community k, and a value of 0 indicates that node v i Not belonging to community k; Setting the embedding dimension to K means that the matrix Z is regarded as a community assignment matrix, i.e., H = Z, and the node embedding is obtained by optimizing the modularity; This application also introduces a learnable fully connected layer F = R h×k Embeddings are mapped to community assignments; here, h represents the number of communities and k represents the desired higher dimensionality; Fully connected layer F = R h×k The role of is to map the low-dimensional representation of each community to a higher-dimensional space, so that the embedding of the node can capture the structure and semantic information of the network. The loss function L that maximizes the modularity is Mod The definition is as follows: The goal of the final model is to minimize the following loss function: L=L Gate -βL Mod Where β represents the hyperparameters of different loss weights; The distinction between real alarms and false alarms is achieved based on the results of community classification. Modularity is used to measure the difference between the internal connections of the community and the internal connections of the same division in a random graph. Real attackers will form highly concentrated subgroups in the network, and the connections between them are frequent and strong, which helps to identify real attack activities, that is, distinguish real alarms.
Citation Information
Cited By
Transform-based network security alarm noise reduction method
CN119449581A
