Network event aggregation method and device based on non-negative matrix factorization

Abstract

The invention discloses a network event aggregation method and device based on non-negative matrix factorization. The network event aggregation method comprises the steps that network flow data are acquired and preprocessed; performing analysis processing on the preprocessed network flow data to generate a basic network event set; clustering the basic network event set through a clustering algorithm, and extracting and generating a key event description set; and constructing a pattern matrix for the key event description set and carrying out matrix iterative decomposition processing to obtain an aggregation event set. According to the invention, event aggregation and network threat discovery can be realized, the network behavior analysis accuracy is effectively improved, and the network security is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a network event aggregation method and device. Background Art

[0002] The diversity and complexity of current network devices are increasing. From traditional desktops and laptops to various IoT devices such as smart phones, smart home devices, and industrial control systems, network-connected terminals are showing a trend of diversity. This makes the network architecture more complex and the management and maintenance of network security more challenging. Network devices generate massive amounts of long-term and long-term discrete traffic data, carrying user information, communication content, and interaction data between network devices. These data exist in the form of network flows and are highly discrete, including combinations of different times, locations, and network protocols. Network management needs to comprehensively consider the temporal and spatial characteristics of the data, as well as the impact of network traffic patterns on security incident detection and response.

[0003] For long-term and spatial discrete network traffic, designing a flexible and effective network security event aggregation method to achieve event extraction and network threat identification of underlying traffic is of great significance to ensuring network security.

[0004] At present, there are three main methods for aggregating network security events: log aggregation is the most basic form of network security event aggregation. The aggregator standardizes the logs of each device for centralized management and further analysis. In a long-term and discrete environment, log aggregation will cause tremendous pressure on performance and storage; SIEM (Security Information and Event Management System) can centrally collect and manage data from multiple devices and programs, provide real-time monitoring and alarm functions, and help organizations detect and respond to security incidents in a timely manner, but its configuration and management require high technical costs and resources, and it cannot process large amounts of log and event data; network traffic analysis can provide in-depth analysis of traffic and identify abnormal behavior in the early stages of an attack, but correspondingly, processing and storing large amounts of real-time data consumes a lot of computing and storage resources.

[0005] Patent CN118364163A discloses a method and system for automatically generating a special report on online public opinion based on event aggregation, including: obtaining all media news to be monitored for public opinion, classifying the full amount of media news, and obtaining the target text belonging to the target category in the full amount of news; extracting the keywords of the target text, and forming a feature text with the keywords and each article in the target text, and obtaining the feature vector of each article through text vectorization processing, clustering the feature vector, clustering the articles belonging to the same event in the target text into an event set, and merging the feature words of all articles in each event set as the label of each event; combining the keywords that users pay attention to and the amount of media coverage of each event, quantifying the labels of all events, and obtaining the importance of each event set, and displaying the events and their labels whose importance is greater than the preset value. It over-relies on the accuracy of the text model. When faced with diversified and dynamically changing news texts, the accuracy of the model will decrease. In addition, the parameter adjustment complexity of the two-level clustering algorithm is relatively high, and a lot of adjustments are required in practical applications.

[0006] Therefore, how to provide a processing method that can perform event extraction and threat identification on long-term spatial and temporal discrete network traffic is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0007] In view of the defects existing in the above-mentioned prior art, the present invention provides a network event aggregation method and device, which can realize event aggregation and network threat discovery, effectively improve the accuracy of network behavior analysis, and ensure network security.

[0008] In a first aspect, the present invention provides a network event aggregation method, comprising:

[0009] Obtain network traffic data and perform preprocessing;

[0010] Analyze and process the pre-processed network traffic data to generate a basic network event set;

[0011] Clustering the basic network event set through clustering algorithm and extracting and generating the key event description set;

[0012] A pattern matrix is ​​constructed for the key event description set and the matrix is ​​iteratively decomposed to obtain an aggregated event set.

[0013] Furthermore, network traffic data is obtained and preprocessed, including:

[0014] Capture network traffic data on a specific network node and generate corresponding data format files;

[0015] Extract key information from the data format file and generate a feature format file containing the key information.

[0016] Furthermore, the pre-processed network traffic data is parsed and processed to generate a basic network event set, including:

[0017] Obtain network security event generation rules, and divide them according to corresponding protocol types to obtain multiple rule template sets;

[0018] Based on the protocol type of the characteristic format file, the characteristic format file is divided to obtain multiple file sets;

[0019] Based on the protocol type, the file set is matched with the rule template set;

[0020] Based on the file key information of the feature format file, each feature format file in the file set is matched with all event generation rules in the corresponding rule template set in sequence, and the feature format file that successfully matches in each file set is given;

[0021] The successfully matched feature format files in each file set generate corresponding events according to the matched event generation rules, and all events are integrated to form a basic network event set.

[0022] Furthermore, the basic network event set is clustered by a clustering algorithm and a key event description set is extracted and generated, including:

[0023] Extract key event information from each event in the basic network event set and generate an event representation vector;

[0024] Clustering the event key information of the event representation vector to obtain multiple sample sets and integrate them to form a cluster set, and at the same time generating a key event description set corresponding to each sample set and being an empty set;

[0025] Perform divergence calculations on the cluster sets in turn to determine the feature scores corresponding to the key information of each event;

[0026] Based on the feature score corresponding to each event key information, determine the relevance value of each event key information, and remove the event key information with the largest relevance value in each sample set from the corresponding sample set, and add it to the key event description set corresponding to the sample set from which the event key information is removed;

[0027] Repeatedly update the correlation value of the remaining key event information in the clustering set based on the feature score corresponding to each key event information, remove the key event information with the largest correlation value in each sample set from the sample set, and add it to the key event description set corresponding to the sample set from which the key event information was removed, until the number of key event information in the key event description set corresponding to each sample set reaches a predetermined number, thereby obtaining a final key event description set.

[0028] Furthermore, the divergence calculation is performed on the cluster sets in turn to determine the feature score corresponding to the key information of each event, including:

[0029] Determine all terms and the number of key event information containing each term based on the cluster set;

[0030] Obtain a target event description set corresponding to network security, and determine the number of key event information containing each term in the target event description set based on all terms in the clustering set;

[0031] Determine the feature score of each term based on the number of key event information of each term in the target event description set and the clustering set;

[0032] Based on the terms contained in each event key information in the cluster set and the feature scores of the corresponding terms, a feature score of each event key information in the cluster set is given.

[0033] Furthermore, the feature score of each term satisfies the following relationship:

[0034]

[0035] In the formula, w(f k ) is the term f k The feature score, f k is the kth term, KL() is the divergence calculation, TD t Describes the target event set, D t is the cluster set, Pr() is the probability function, tf td (f k ) is the target event description set containing the term f k The number of key event information, tf d (f k ) is the cluster set containing term f k The number of key event information, |TD t | is the number of key event information that contains at least one term in the target event description set, |D t | is the number of key information of events containing terms in the cluster set;

[0036] The feature score of each event key information in the cluster set satisfies the following relationship:

[0037]

[0038] In the formula, w(q e,i ) is the key information of the event q e,i The feature score, q e,i is the key information of the ith event.

[0039] Furthermore, based on the feature score corresponding to each event key information, the relevance value of each event key information is determined to satisfy the following relationship:

[0040]

[0041] Where, MMR i Key event information e,i The correlation value, n i Key event information e,i The size of the sample set, n max is the size of the largest sample set in the cluster set;

[0042] Update the relevance value of the remaining event key information of the cluster set to satisfy the following relationship:

[0043]

[0044] Where, MMR j The remaining event key information q is the clustering set e,j The correlation value, σ is the weight parameter, w(q e,j ) is the key information of the event q e,j feature score, RC is the key event description set, and sim() is the cosine similarity function.

[0045] Furthermore, a pattern matrix is ​​constructed for the key event description set and the matrix is ​​iteratively decomposed to obtain an aggregated event set, including:

[0046] Perform vector processing on each key event description set to obtain a pattern vector;

[0047] Perform weighted processing on all pattern vectors to obtain a weighted pattern matrix;

[0048] Decomposing the weighted pattern matrix based on non-negative matrix factorization to obtain a decomposition function containing a low-rank factor matrix;

[0049] Based on the hierarchical alternating least squares method of block coordinate descent, the update rule of the low-rank factor matrix is ​​determined and the low-rank factor matrix is ​​updated until the termination condition is met, and a set of aggregation events is given.

[0050] Furthermore, the weighted pattern matrix satisfies the following relationship:

[0051] W sum =[n1E sum,1 ,n2E sum,2 ,…,n k E sum,k ]

[0052] Where W sumis the weighted pattern matrix, E sum,k is the kth pattern vector, n k The number of event key information corresponding to a specific target event in the kth key event description set;

[0053] Decomposition function satisfies the following relationship:

[0054]

[0055] Where W S and H are both low-rank factor matrices, W S is an M×|RC sum | matrix, each column represents an aggregate event, M is the total number of key event information in the key event description set; H is a |RC|H t ×|RC sum |Matrix, representing W S Each key event description set of aggregated events with different similarities in , T is the transpose of the matrix, ║·║ F is the Frobenius norm of the matrix, λ is the regularization parameter, ║·║ 1,1 is the L1 norm, RC is the key event description set, H t The number of batches describing the set of key events identified from the earliest historical timestamp to the current timestamp, RC sum is a collection of aggregated events;

[0056] Update the rules to satisfy the following relationship:

[0057]

[0058] Where W S (l) is W S The lth column vector of H(l) is the lth column vector of H, (W sum H)(l) is W sum The lth column vector of H, (·)(l) is the lth column vector of the matrix, (·) ll is the element in the lth row and lth column of the matrix, 1 M is a vector of all 1s with dimension M, The dimension is |RC|H t All 1 vectors, (t) and (t+1) are before and after the update, respectively. For example, H (t) (l) is H(l) before updating.

[0059] In a second aspect, the present invention further provides a network event aggregation device, which adopts the above-mentioned network event aggregation method, and the network event aggregation device includes:

[0060] A data acquisition module, which is used to acquire network traffic data and perform preprocessing;

[0061] A parsing and processing module, which is used to parse and process the pre-processed network traffic data to generate a basic network event set;

[0062] A clustering extraction module, which is used to cluster the basic network event set through a clustering algorithm and extract and generate a key event description set;

[0063] The matrix decomposition module is used to construct a pattern matrix for the key event description set and perform matrix iterative decomposition processing to obtain an aggregated event set.

[0064] Furthermore, the data acquisition module also includes:

[0065] Capture network traffic data on a specific network node and generate corresponding data format files;

[0066] Extract key information from the data format file and generate a feature format file containing the key information.

[0067] Furthermore, the analysis processing module also includes:

[0068] Obtain network security event generation rules, and divide them according to corresponding protocol types to obtain multiple rule template sets;

[0069] Based on the protocol type of the characteristic format file, the characteristic format file is divided to obtain multiple file sets;

[0070] Based on the protocol type, the file set is matched with the rule template set;

[0071] Based on the file key information of the feature format file, each feature format file in the file set is matched with all event generation rules in the corresponding rule template set in sequence, and the feature format file that successfully matches in each file set is given;

[0072] The successfully matched feature format files in each file set generate corresponding events according to the matched event generation rules, and all events are integrated to form a basic network event set.

[0073] Furthermore, the cluster extraction module includes:

[0074] Extract key event information from each event in the basic network event set and generate an event representation vector;

[0075] Clustering the event key information of the event representation vector to obtain multiple sample sets and integrate them to form a cluster set, and at the same time generating a key event description set corresponding to each sample set and being an empty set;

[0076] Perform divergence calculations on the cluster sets in turn to determine the feature scores corresponding to the key information of each event;

[0077] Based on the feature score corresponding to each event key information, determine the relevance value of each event key information, and remove the event key information with the largest relevance value in each sample set from the corresponding sample set, and add it to the key event description set corresponding to the sample set from which the event key information is removed;

[0078] Repeatedly update the correlation value of the remaining key event information in the clustering set based on the feature score corresponding to each key event information, remove the key event information with the largest correlation value in each sample set from the sample set, and add it to the key event description set corresponding to the sample set from which the key event information was removed, until the key event information in the key event description set corresponding to each sample set reaches a predetermined number, thereby obtaining a final key event description set.

[0079] Furthermore, the cluster extraction module also includes:

[0080] Determine all terms and the number of key event information containing each term based on the cluster set;

[0081] Obtain a target event description set corresponding to network security, and determine the number of key event information containing each term in the target event description set based on all terms in the clustering set;

[0082] Determine the feature score of each term based on the number of key event information of each term in the target event description set and the clustering set;

[0083] Based on the terms contained in each event key information in the cluster set and the feature scores of the corresponding terms, a feature score of each event key information in the cluster set is given.

[0084] Furthermore, the matrix decomposition module also includes:

[0085] Perform vector processing on each key event description set to obtain a pattern vector;

[0086] Perform weighted processing on all pattern vectors to obtain a weighted pattern matrix;

[0087] Decomposing the weighted pattern matrix based on non-negative matrix factorization to obtain a decomposition function containing a low-rank factor matrix;

[0088] Based on the hierarchical alternating least squares method of block coordinate descent, the update rule of the low-rank factor matrix is ​​determined and the low-rank factor matrix is ​​updated until the termination condition is met, and a set of aggregation events is given.

[0089] The present invention provides a network event aggregation method and device, which have at least the following beneficial effects:

[0090] (1) The present invention can sequentially acquire, parse, cluster, extract, and iteratively decompose long-term and spatially discrete network traffic data, realize event aggregation and network threat discovery, effectively improve the accuracy of network behavior analysis, and ensure network security.

[0091] (2) Formalizing basic network events into event representation vectors can fully record event-related detailed information and key features, thereby improving the accuracy of subsequent event aggregation processes.

[0092] (3) It can effectively utilize the spatiotemporal characteristics of network events, capture key features, and process large-scale data sets. It solves the problem that traditional methods cannot fully aggregate network events with large timestamp gaps, and effectively improves the network event aggregation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A flowchart of a network event aggregation method provided by the present invention;

[0094] Figure 2 A flowchart of generating a basic network event set provided by an embodiment of the present invention;

[0095] Figure 3 A flowchart of generating a key event description set provided by an embodiment of the present invention;

[0096] Figure 4 A flow chart for determining feature scores provided for an embodiment of the present invention;

[0097] Figure 5 A flowchart of obtaining an aggregated event set provided by an embodiment of the present invention;

[0098] Figure 6 A schematic diagram of a network event aggregation device provided by the present invention. DETAILED DESCRIPTION

[0099] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0100] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0101] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0102] like Figure 1 As shown, the present invention provides a network event aggregation method, which may include:

[0103] Obtain network traffic data and perform preprocessing;

[0104] Analyze and process the pre-processed network traffic data to generate a basic network event set;

[0105] Clustering the basic network event set through clustering algorithm and extracting and generating the key event description set;

[0106] A pattern matrix is ​​constructed for the key event description set and the matrix is ​​iteratively decomposed to obtain an aggregated event set.

[0107] The network traffic data acquired by the present invention comes from a real environment, and network events are aggregated for long-term and spatial discrete traffic data to identify network threats, which can effectively improve the security performance of network terminals; the network event aggregation method of the present invention can effectively utilize the spatiotemporal characteristics of events, capture key features, and solve the problem that traditional methods cannot fully aggregate network events with large timestamp differences, effectively improving the network event aggregation capability. The basic network events parsed from network traffic are aggregated through a clustering algorithm and an event aggregation algorithm using non-negative matrix decomposition, thereby improving the accuracy of network behavior analysis.

[0108] Obtain network traffic data and perform preprocessing, including:

[0109] Capture network traffic data on a specific network node and generate corresponding data format files;

[0110] Extract key information from the data format file and generate a feature format file containing the key information.

[0111] In actual application scenarios, obtaining network traffic data and preprocessing it may include:

[0112] Select a suitable packet capture tool, install it and configure it, start capturing traffic on a specific network node, and save the captured data as a pacp file (data format file);

[0113] Perform feature extraction on the pcap file, use Python to extract key file information such as URL, IP address, port number, HTTP header, etc. from the captured traffic file, and generate a csv file (feature format file) including the key file information.

[0114] The present invention realizes feature extraction by capturing original network flow data and performing preprocessing, which is beneficial to the generation of a basic network event set.

[0115] like Figure 2 As shown, after preprocessing, the preprocessed network traffic data is parsed to generate a basic network event set, which may include:

[0116] Obtain network security event generation rules, and divide them according to corresponding protocol types to obtain multiple rule template sets;

[0117] Based on the protocol type of the characteristic format file, the characteristic format file is divided to obtain multiple file sets;

[0118] Based on the protocol type, the file set is matched with the rule template set;

[0119] Based on the file key information of the feature format file, each feature format file in the file set is matched with all event generation rules in the corresponding rule template set in sequence, and the feature format file that successfully matches in each file set is given;

[0120] The successfully matched feature format files in each file set generate corresponding events according to the matched event generation rules, and all events are integrated to form a basic network event set.

[0121] In actual application scenarios, event generation rules are divided into four rule template sets of TCP, UDP, ICMP, and IP according to the protocol type of the traffic protocol and represented by a linked list. The data is assigned to a pre-determined subset of event generation rules according to the protocol type of the current data packet (feature format file); the data segment of the feature format file is checked for content to determine whether each rule in the corresponding event generation rule subset conforms to the rule of the data packet (feature format file); if the match is successful, the feature format file generates a corresponding event, otherwise the next rule is matched until the traversal of the event generation rule table is completed.

[0122] Among them, the event generation rule base is pre-defined based on known abnormal behaviors or attack patterns and other information, which may include specific protocol fields, specific traffic characteristics or payload content, etc.;

[0123] Based on the file key information of the feature format file, each feature format file in the file set is matched with all event generation rules in the corresponding rule template set in sequence, including:

[0124] Extract key file information of the data segment from the feature format file, which may be the actual data in the application layer protocol or the control information in the protocol header. For example, for an HTTP request, the content that may be extracted includes the request method, URL, request parameters, etc. For a TCP packet, the sequence number, flag bit, etc. may be extracted;

[0125] Compare the key file information of the data segment with the conditions in the event generation rule library to determine whether the triggering conditions of the rule are met, including field matching, pattern matching, statistical feature matching, etc.

[0126] The present invention performs protocol analysis on the pre-processed network traffic data, performs rule matching according to event generation rules, generates a corresponding basic network event set, and provides reliable data guarantee for event aggregation.

[0127] like Figure 3 As shown, after generating the basic network event set, the basic network event set can be clustered by a clustering algorithm and a key event description set can be extracted and generated, including:

[0128] For each event in the basic network event set, the key event information is extracted and an event representation vector is generated, which satisfies the following relationship:

[0129] ERV=(ID,Q e ,date,type)

[0130] Where ERV is the event representation vector, ID is the unique identification number of each event, and Q e Represents a set of key domain-related patterns of the same event. The key domain-related patterns represent the description of network events, including the attack target, the attacker, the current operation, the tools used, etc. Date represents a single timestamp or a set of timestamps when an event occurs. Type represents the event category, for example, data leakage, account hijacking, DDoS, malware attack, etc., to which the attacker's current behavior belongs. The basic network event set is formalized into an event representation vector, which fully records the detailed information and key features related to the event, providing effective data assurance for the model. Among them, the key information of the event includes key domain-related patterns.

[0131] Clustering the event key information of the event representation vector to obtain multiple sample sets and integrate them to form a cluster set, and at the same time generating a key event description set corresponding to each sample set and which is an empty set; wherein the key event description set generated in this step is an empty set, which is supplemented by a subsequent event key information adding process;

[0132] Perform divergence calculations on the cluster sets in turn to determine the feature scores corresponding to the key information of each event;

[0133] Based on the feature score corresponding to each event key information, determine the relevance value of each event key information, and remove the event key information with the largest relevance value in each sample set from the corresponding sample set, and add it to the key event description set corresponding to the sample set from which the event key information is removed;

[0134] Repeatedly update the correlation value of the remaining key event information in the clustering set based on the feature score corresponding to each key event information, remove the key event information with the largest correlation value in each sample set from the sample set, and add it to the key event description set corresponding to the sample set from which the key event information was removed, until the key event information in the key event description set corresponding to each sample set reaches a predetermined number, thereby obtaining a final key event description set.

[0135] In actual application scenarios, after generating the event representation vector, the following steps are also required:

[0136] The input of the representative example set (key event description set) extraction process includes a cluster set C and the corresponding sample set C e = {q e,1 ,…,q e,k}, and the number of events n in the i-th cluster i ; Among them, the Affinity Propagation Clustering Algorithm is used to divide the set of event key information of all event representation vectors into k clusters. The Affinity Propagation Clustering Algorithm determines the number of clusters according to the actual points in the data set. The clustering result is a cluster set C = (C1, C2,,…, C k );

[0137] Use KL divergence to calculate the sample set C e Each instance q e,i Feature score of (key event information);

[0138] Calculate the MMR of each instance in the sample set i , and select the one with the highest MMR i Instance of value q e,iAdd the representative example set RC (critical incident description set);

[0139] For each instance q in the sample set e,j ∈C e \q e,i , calculate its MMR j and select the one with the highest MMR j Instance of value q e,j Join the representative example set RC;

[0140] The above instance selection process is repeated until the number of selected instances reaches the specified number.

[0141] The present invention extracts key domain related patterns (key event information) of each event and connects them in sequence to generate a composite pattern, divides the composite pattern set into k clusters using a clustering algorithm, and extracts representative examples from each cluster as a summary description of the network event.

[0142] like Figure 4 As shown in FIG. 1 , after obtaining the cluster set, the divergence calculation can be performed on the cluster set in sequence to determine the feature score corresponding to each event key information, including:

[0143] Determine all terms and the number of key event information containing each term based on the cluster set;

[0144] Obtain a target event description set corresponding to network security, and determine the number of key event information containing each term in the target event description set based on all terms in the clustering set;

[0145] Determine the feature score of each term based on the number of key event information of each term in the target event description set and the clustering set;

[0146] Based on the terms contained in each event key information in the cluster set and the feature scores of the corresponding terms, a feature score of each event key information in the cluster set is given.

[0147] Among them, the feature score of each term satisfies the following relationship:

[0148]

[0149] In the formula, w(f k ) is the term f k The feature score, f k is the kth term, KL() is the divergence calculation, TD t Describes the target event set, D t is the cluster set, Pr() is the probability function, tf td (f k) is the target event description set containing the term f k The number of key event information, tf d (f k ) is the cluster set containing term f k The number of key event information, |TD t | is the number of key event information that contains at least one term in the target event description set, |D t | is the number of key information of events containing terms in the cluster set;

[0150] The feature score of each event key information in the cluster set satisfies the following relationship:

[0151]

[0152] In the formula, w(q e,i ) is the key information of the event q e,i The feature score, q e,i is the key information of the ith event.

[0153] The present invention adopts a public network security corpus (target event description set), uses KL divergence to calculate the probability distribution for each term in an instance, and takes the sum of the probability distributions of all terms in the instance as the feature score of the instance.

[0154] Based on the feature score corresponding to each event key information, the relevance value of each event key information is determined to satisfy the following relationship:

[0155]

[0156] Where, MMR i Key event information e,i The correlation value, n i Key event information e,i The size of the sample set, n max is the size of the largest sample set in the cluster set;

[0157] Update the relevance value of the remaining event key information of the cluster set to satisfy the following relationship:

[0158]

[0159] Where, MMR j The remaining event key information q is the clustering set e,j The correlation value of the score item and the similarity item are weight parameters, which are used to balance the contribution of the score item and the similarity item. It is determined according to the specific situation. In this case, since the similarity of the key event description (key event information) extracted from each cluster is of high importance, it can be set between 0.7 and 0.8.e,j ) is the key information of the event q e,j feature score, RC is the key event description set, sim() is the cosine similarity function, Sim(q e,i ,q e,j ) represents the key information of the event q e,i With q e,j This embodiment calculates the MMR of each instance in the sample set according to the algorithm formula i , providing data support for selecting representative examples of each cluster. Calculate each instance q in the sample set that does not belong to RC e,j MMR j , in order to select representative examples from each cluster.

[0160] Among them, two key event information q e,i With q e,j The cosine similarity between satisfies the following relationship:

[0161]

[0162] In the formula, and They are respectively the key information of the event q e,i and key event information e,j The average word embedding of and Respectively, vector and The mold length.

[0163] The present invention performs clustering processing on the basic network event set through a clustering algorithm, groups similar events according to the key information of the event, and can classify events with large timestamp differences into the same category, breaking through the limitations of traditional methods in processing events with long time intervals. Through matrix iterative decomposition, discrete network events with large timestamp differences can be effectively aggregated together to generate the final aggregated event. The aggregation effect of traditional methods on long-time and space events is significantly improved, and the accuracy of network threat detection and response is enhanced. At the same time, using iterative optimization technology, high-dimensional data can be reduced in dimensionality, which improves the accuracy and efficiency of event aggregation and helps solve the problem of processing large-scale data sets.

[0164] like Figure 5 As shown, after obtaining the final key event description set, a pattern matrix can be constructed for the key event description set and the matrix can be iteratively decomposed to obtain an aggregated event set, including:

[0165] Perform vector processing on each key event description set to obtain a pattern vector;

[0166] Perform weighted processing on all pattern vectors to obtain a weighted pattern matrix;

[0167] Decomposing the weighted pattern matrix based on non-negative matrix factorization to obtain a decomposition function containing a low-rank factor matrix;

[0168] Based on the hierarchical alternating least squares method of block coordinate descent, the update rule of the low-rank factor matrix is ​​determined and the low-rank factor matrix is ​​updated until the termination condition is met, and a set of aggregation events is given.

[0169] The present invention constructs a pattern vector representation for each key event description set, uses non-negative matrix decomposition to decompose the weighted pattern matrix into two low-rank factor matrices, and iteratively updates to obtain aggregated events, effectively utilizing the temporal and spatial characteristics of network events and capturing key features. This solves the problem that traditional methods cannot fully aggregate network events with large timestamp gaps, and effectively improves the network event aggregation capability.

[0170] Among them, the weighted pattern matrix satisfies the following relationship:

[0171] W sum =[n1E sum,1 ,n2E sum,2 ,…,n k E sum,k ]

[0172] Where W sum is the weighted pattern matrix, E sum,k is the kth pattern vector, which is an M-dimensional column vector, n k The number of event key information corresponding to a specific target event in the kth key event description set;

[0173] Decomposition function satisfies the following relationship:

[0174]

[0175] Where W S and H are both low-rank factor matrices, W S is an M×|RC sum | matrix, each column represents an aggregate event, M is the total number of key event information in the key event description set; H is a |RC|H t ×|RC sum |Matrix, representing W S Each key event description set of aggregated events with different similarities in , T is the transpose of the matrix, ║·║ F is the Frobenius norm of the matrix, is a square term of the Frobenius norm, representing the difference between the two matrices, λ is the regularization parameter, ║·║ 1,1is the L1 norm, which can control W S and the sparsity of H, RC is the key event description set, H t is the number of batches of key event description sets identified from the earliest historical timestamp to the current timestamp. The batches are divided according to a certain time interval, and each batch is matched to identify the events corresponding to the basic network event set. RC sum A collection of aggregate events.

[0176] The present invention expresses the weighted pattern matrix as the product form of low-rank factor matrices through non-negative matrix decomposition, which helps to reduce the dimension of data, improve the efficiency of data expression, and can retain the main features and structure of the original data, thereby improving the accuracy of event aggregation.

[0177] Update the rules to satisfy the following relationship:

[0178]

[0179] Where W S (l) is W S The lth column vector of H(l) is the lth column vector of H, (W sum H)(l) is W sum The lth column vector of H, (·)(l) is the lth column vector of the matrix, (·) ll Extract the specific value in the matrix for the element in the lth row and lth column of the matrix, 1 M is a vector of all 1s with dimension M, The dimension is |RC|H t The all-1 vector, (t) and (t+1) are before and after the update respectively.

[0180] Update the low-rank factor matrix until the termination condition is met. S The update changes of W and H are very small, that is, between two consecutive iterations S When the change of and H is lower than a certain small threshold, the termination condition is met and the iteration stops. and ||H (t+1) -H (t) ||<ε, ε is a very small integer.

[0181] The present invention collects raw network traffic data from discrete multi-dimensional terminals and processes it into a basic network event set. The events in the basic network event set are formalized into a unified data structure, and their key features and detailed information are fully recorded. Network events are divided into multiple different pattern clusters through clustering and representative examples are selected. Each network event is represented by a pattern matrix, and feature extraction and dimensionality reduction are performed through non-negative matrix decomposition. Event clustering is finally achieved through iterative optimization according to update rules.

[0182] like Figure 6 As shown, the present invention also provides a network event aggregation device, which adopts the above network event aggregation method, and the network event aggregation device includes:

[0183] A data acquisition module, which is used to acquire network traffic data and perform preprocessing;

[0184] A parsing and processing module, which is used to parse and process the pre-processed network traffic data to generate a basic network event set;

[0185] A clustering extraction module, which is used to cluster the basic network event set through a clustering algorithm and extract and generate a key event description set;

[0186] The matrix decomposition module is used to construct a pattern matrix for the key event description set and perform matrix iterative decomposition processing to obtain an aggregated event set.

[0187] Furthermore, the data acquisition module also includes:

[0188] Capture network traffic data on a specific network node and generate corresponding data format files;

[0189] Extract key information from the data format file and generate a feature format file containing the key information.

[0190] Furthermore, the analysis processing module also includes:

[0191] Obtain network security event generation rules, and divide them according to corresponding protocol types to obtain multiple rule template sets;

[0192] Based on the protocol type of the characteristic format file, the characteristic format file is divided to obtain multiple file sets;

[0193] Based on the protocol type, the file set is matched with the rule template set;

[0194] Based on the file key information of the feature format file, each feature format file in the file set is matched with all event generation rules in the corresponding rule template set in sequence, and the feature format file that successfully matches in each file set is given;

[0195] The successfully matched feature format files in each file set generate corresponding events according to the matched event generation rules, and all events are integrated to form a basic network event set.

[0196] Furthermore, the cluster extraction module includes:

[0197] Extract key event information from each event in the basic network event set and generate an event representation vector;

[0198] Clustering the event key information of the event representation vector to obtain multiple sample sets and integrate them to form a cluster set, and at the same time generating a key event description set corresponding to each sample set and being an empty set;

[0199] Perform divergence calculations on the cluster sets in turn to determine the feature scores corresponding to the key information of each event;

[0200] Based on the feature score corresponding to each event key information, determine the relevance value of each event key information, and remove the event key information with the largest relevance value in each sample set from the corresponding sample set, and add it to the key event description set corresponding to the sample set from which the event key information is removed;

[0201] Repeatedly update the correlation value of the remaining key event information in the clustering set based on the feature score corresponding to each key event information, remove the key event information with the largest correlation value in each sample set from the sample set, and add it to the key event description set corresponding to the sample set from which the key event information was removed, until the key event information in the key event description set corresponding to each sample set reaches a predetermined number, thereby obtaining a final key event description set.

[0202] Furthermore, the cluster extraction module also includes:

[0203] Determine all terms and the number of key event information containing each term based on the cluster set;

[0204] Obtain a target event description set corresponding to network security, and determine the number of key event information containing each term in the target event description set based on all terms in the clustering set;

[0205] Determine the feature score of each term based on the number of key event information of each term in the target event description set and the clustering set;

[0206] Based on the terms contained in each event key information in the cluster set and the feature scores of the corresponding terms, a feature score of each event key information in the cluster set is given.

[0207] Furthermore, the matrix decomposition module also includes:

[0208] Perform vector processing on each key event description set to obtain a pattern vector;

[0209] Perform weighted processing on all pattern vectors to obtain a weighted pattern matrix;

[0210] Decomposing the weighted pattern matrix based on non-negative matrix factorization to obtain a decomposition function containing a low-rank factor matrix;

[0211] Based on the hierarchical alternating least squares method of block coordinate descent, the update rule of the low-rank factor matrix is ​​determined and the low-rank factor matrix is ​​updated until the termination condition is met, and a set of aggregation events is given.

[0212] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A network event aggregation method based on non-negative matrix decomposition, characterized in that: include: Obtain network traffic data and perform preprocessing; Analyze and process the pre-processed network traffic data to generate a basic network event set; Clustering the basic network event set through clustering algorithm and extracting and generating the key event description set; A pattern matrix is ​​constructed for the key event description set and the matrix is ​​iteratively decomposed to obtain an aggregated event set.

2. The network event aggregation method according to claim 1, characterized in that: The obtaining of network traffic data and preprocessing further includes: Capture network traffic data on a specific network node and generate corresponding data format files; Extract key file information from the data format file to generate a feature format file containing the key file information.

3. The network event aggregation method according to claim 2, characterized in that: The parsing and processing of the pre-processed network traffic data to generate a basic network event set further includes: Obtain network security event generation rules, and divide them according to corresponding protocol types to obtain multiple rule template sets; Based on the protocol type of the characteristic format file, the characteristic format file is divided to obtain multiple file sets; Based on the protocol type, the file set is matched with the rule template set; Based on the file key information of the feature format file, each feature format file in the file set is matched with all event generation rules in the corresponding rule template set in sequence, and the feature format file that successfully matches in each file set is given; The successfully matched feature format files in each file set generate corresponding events according to the matched event generation rules, and all events are integrated to form a basic network event set.

4. The network event aggregation method according to claim 1 or 3, characterized in that: The method of clustering the basic network event set by a clustering algorithm and extracting and generating a key event description set further includes: Extract key event information from each event in the basic network event set and generate an event representation vector; Clustering the event key information of the event representation vector to obtain multiple sample sets and integrating them to form a cluster set, and generating a key event description set corresponding to each sample set and being an empty set; Perform divergence calculation on the cluster sets in turn to determine the feature score corresponding to each event key information; Based on the feature score corresponding to each event key information, determine the relevance value of each event key information, and remove the event key information with the largest relevance value in each sample set from the corresponding sample set, and add it to the key event description set corresponding to the sample set from which the event key information is removed; Repeatedly update the correlation value of the remaining key event information in the clustering set based on the feature score corresponding to each key event information, remove the key event information with the largest correlation value in each sample set from the sample set, and add it to the key event description set corresponding to the sample set from which the key event information was removed, until the number of key event information in the key event description set corresponding to each sample set reaches a predetermined number, thereby obtaining a final key event description set.

5. The network event aggregation method according to claim 4, characterized in that: The divergence calculation is performed on the cluster set in sequence to determine the feature score corresponding to each event key information, further comprising: Determine all terms and the number of key event information containing each term based on the cluster set; Obtain a target event description set corresponding to network security, and determine the number of key event information containing each term in the target event description set based on all terms in the clustering set; Determine the feature score of each term based on the number of key event information of each term in the target event description set and the clustering set; Based on the terms contained in each event key information in the cluster set and the feature scores of the corresponding terms, a feature score of each event key information in the cluster set is given.

6. The network event aggregation method according to claim 5, characterized in that: The feature score of each term satisfies the following relationship: In the formula, w(f k ) is the term f k The feature score, f k is the kth term, KL() is the divergence calculation, TD t Describes the target event set, D t is the cluster set, Pr() is the probability function, tf td (f k ) is the target event description set containing the term f k The number of key event information, tf d (f k ) is the cluster set containing term f k The number of key event information, |TD t | is the number of key event information that contains at least one term in the target event description set, |D t | is the number of key information of events containing terms in the cluster set; The feature score of each event key information in the cluster set satisfies the following relationship: In the formula, w(q e,i ) is the key information of the event q e,i The feature score, q e,i is the key information of the ith event.

7. The network event aggregation method according to claim 6, characterized in that: The correlation value of each event key information is determined based on the feature score corresponding to each event key information, satisfying the following relationship: Where, MMR i Key event information e,i The correlation value, n i Key event information e,i The size of the sample set, n max is the size of the largest sample set in the cluster set; The relevance value of the key information of the remaining events of the updated cluster set satisfies the following relationship: Where, MMR j The remaining event key information q of the cluster set e,j The correlation value, σ is the weight parameter, w(q e,j ) is the key information of the event q e,j feature score, RC is the key event description set, and sim() is the cosine similarity function.

8. The network event aggregation method according to claim 1 or 7, characterized in that: The step of constructing a pattern matrix for the key event description set and performing matrix iterative decomposition processing to obtain an aggregated event set further includes: Perform vector processing on each key event description set to obtain a pattern vector; Perform weighted processing on all pattern vectors to obtain a weighted pattern matrix; Decomposing the weighted pattern matrix based on non-negative matrix decomposition to obtain a decomposition function including a low-rank factor matrix; Based on the hierarchical alternating least squares method of block coordinate descent, the update rule of the low-rank factor matrix is ​​determined and the low-rank factor matrix is ​​updated until the termination condition is met, and a set of aggregation events is given.

9. The network event aggregation method according to claim 8, characterized in that: The weighted pattern matrix satisfies the following relationship: W sum [n1E sum,1 ,n2E sum,2 ,…,n k E sum,k ] Where W sum is the weighted pattern matrix, E sum,k is the kth pattern vector, n k The number of event key information corresponding to a specific target event in the kth key event description set; The decomposition function satisfies the following relationship: Where W S and H are both low-rank factor matrices, W S is an M×|RC sum | matrix, each column represents an aggregate event, M is the total number of key event information in the key event description set; H is a |RC|H t ×|RC sum |Matrix, representing W S Each key event description set of aggregated events with different similarities in , T is the transpose of the matrix, ║·║ F is the Frobenius norm of the matrix, λ is the regularization parameter, ║·║ 1,1 is the L1 norm, RC is the key event description set, H t The number of batches describing the set of key events identified from the earliest historical timestamp to the current timestamp, RC sum is a collection of aggregated events; Update the rules to satisfy the following relationship: Where W S (l) is W S The lth column vector of H(l) is the lth column vector of H, (W sum H)(l) is W sum The lth column vector of H, (·)(l) is the lth column vector of the matrix, (·) ll is the element in the lth row and lth column of the matrix, 1 M is a vector of all 1s with dimension M, The dimension is |RC|H t The all-1 vector, (t) and (t+1) are before and after the update respectively.

10. A network event aggregation device based on non-negative matrix decomposition, characterized in that: According to any one of claims 1 to 9, the network event aggregation method based on non-negative matrix decomposition is adopted, and the network event aggregation device comprises: A data acquisition module, which is used to acquire network traffic data and perform preprocessing; The parsing and processing module is used to parse and process the pre-processed network traffic data to generate a basic network event set; A clustering extraction module, which is used to cluster the basic network event set through a clustering algorithm and extract and generate a key event description set; The matrix decomposition module is used to construct a pattern matrix for the key event description set and perform matrix iterative decomposition processing to obtain an aggregated event set.

Images