A multi-dimensional feature real-time extraction method for high-speed network traffic

Abstract

The application relates to a multi-dimensional feature real-time extraction method for high-speed network flow, and belongs to the technical field of network security. In view of the technical difficulty of real-time extraction of multi-dimensional flow features of intelligent network threat detection in a high-speed network environment, the application combines the damping increment statistics and sketch two ideas, proposes an incremental statistics sketch ISS architecture, and based on the architecture, proposes a flow feature real-time extraction method in a high-speed network, which is used for real-time extraction of multi-dimensional flow features in a high-load network. The extraction method has high-speed flow processing capacity and flow feature real-time calculation capacity, and can realize real-time extraction of multi-dimensional flow features in a high-speed network environment.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, specifically relating to a method for real-time extraction of multi-dimensional features of high-speed network traffic. Background Technology

[0002] In recent years, with the rapid development of intelligent technologies in many fields, machine learning-based malicious traffic detection, which improves detection accuracy and can detect encrypted and unknown traffic, has become a research hotspot in the field of network security. Traffic feature extraction is a key step in intelligent malicious traffic detection. Existing traffic feature extraction methods generally classify features into three categories: content features, data flow statistical features, and network connection behavior features. Content features are mainly obtained from unique values ​​in traffic protocol segments and special character sequences in the payload. Data flow statistical features and network connection behavior features are obtained by statistically analyzing collected network traffic data and can be collectively referred to as statistical features.

[0003] To improve the accuracy of malicious traffic detection, existing research on traffic feature extraction methods largely focuses on finding the most effective traffic features, while neglecting the computational complexity of the feature extraction process. However, in real-world backbone network scenarios, due to limitations in computing and storage resources, traditional feature extraction methods struggle to extract features from high-speed traffic in real time, leading to decreased traffic detection accuracy or delayed detection results. Therefore, it is necessary to research traffic feature extraction methods for high-speed networks.

[0004] Damped incremental statistics is a method used to improve feature extraction efficiency. It reduces the time complexity of feature computation through incremental calculation. However, this method cannot address the impact of massive data volumes in high-speed networks on the addressing time required for traffic feature retrieval and storage. Sketching is a method for handling high-speed network traffic measurement problems; existing research has only used sketches to extract simple statistical features, without studying multidimensional traffic feature extraction. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The technical problem to be solved by this invention is: how to address the technical difficulty of real-time extraction of multi-dimensional traffic features in high-speed network environments for intelligent network threat detection, and propose a method for real-time extraction of traffic features in high-speed networks, which can be used to extract multi-dimensional traffic features in real time under high-load network conditions.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, this invention provides a method for real-time extraction of multi-dimensional features from high-speed network traffic, comprising the following steps:

[0009] Step 1: Design the Feature Increment Statistical Sketch (ISS) by combining the damping increment statistics and the sketch data structure;

[0010] Step 2: Real-time extraction of traffic features based on ISS.

[0011] Preferably, in step one, the ISS is designed to consist of the following four parts: packet parsing unit PPU, address mapping unit AMU, sketch storage unit SSU, and feature aggregation unit FAU;

[0012] The PPU is used to parse traffic data packets. For each input data packet, it parses out the corresponding flow ID information and the information of each protocol field. The flow ID refers to the five-tuple information: source IP address, destination IP address, source port number, destination port number, and protocol type. The five-tuple information can be quantized and used as the input of a hash function.

[0013] The AMU consists of multiple sets of hash functions, used to represent the mapping from stream ID to the Sketch Unit (SSU) storage address. The number of hash functions in the AMU is represented as N×M, where N is the number of hash function sets and M is the number of hash functions in each set. Each hash function set consists of M different hash functions from the same hash function family. The number of hash function sets N in the AMU is the number of incremental statistics, and M is used to control the upper bound of the error of the incremental statistics. Incremental statistics refer to the intermediate statistics used to calculate the final statistics in the incremental statistics.

[0014] The SSU consists of multiple hash tables used to store incremental statistics. The multiple hash tables in the SSU are represented as a three-dimensional matrix of size N×M×L, where N is the number of hash tables, M is the number of hash tables in each group, and L is the length of each hash table. The hash address of each hash table in the SSU is calculated using the hash function of the corresponding group in the AMU.

[0015] A set of hash tables in SSU and the corresponding hash functions in AMU form a sketch of shape M×L, with N sketches representing different incremental statistics.

[0016] FAU is implemented by a set of aggregation functions, which are used to aggregate the feature data in SSU, i.e., the incremental statistics, to obtain the final feature values.

[0017] Preferably, in the PPU, the protocol type can be numerically represented and concatenated with information such as source IP address, destination IP address, source port number, and destination port number to obtain the flow ID.

[0018] Preferably, the PPU's packet parsing function can be implemented using one of the underlying libraries, PcapPlusPlus or tshark.

[0019] Preferably, the family of hash functions uses k-global hash functions, denoted as:

[0020]

[0021] Where r represents a prime number, a i and b i A is an integer randomly selected from [0, r-1]. i ≠0, L is the length of the hash table.

[0022] Preferably, the aggregation function includes three types: vertical aggregation within a sketch, vertical aggregation between sketches, and horizontal aggregation between sketches.

[0023] Vertical aggregation within a sketch refers to the process of calculating the estimated values ​​of the incremental statistics corresponding to all flow IDs for each sketch, forming an array of incremental statistics of length L.

[0024] Vertical aggregation between sketches refers to the process of combining different incremental statistics of the same flow ID to obtain the final single flow feature after vertical aggregation of all sketches. Single flow feature refers to the flow feature calculated from all flow data with the same flow ID.

[0025] Horizontal aggregation between sketches refers to the process of calculating the same incremental statistics and flow characteristics of different flow IDs by combining the array of incremental statistics obtained after vertical aggregation within all sketches and the single flow flow characteristics obtained by vertical aggregation between sketches; multi-flow characteristics refer to flow characteristics other than single flow characteristics calculated from all flow data.

[0026] Preferably, step two specifically includes the following steps:

[0027] (1) Obtain a data packet from the raw data stream;

[0028] (2) Parse the data packets in the PPU to obtain the flow ID and information of each protocol field;

[0029] (3) Obtain the N×M hash addresses corresponding to the stream ID in the AMU;

[0030] (4) Based on the hash address obtained in step (3), store the protocol field information of the data packet into the corresponding sketch in SSU and update the incremental statistics.

[0031] (5) Repeat steps (1) to (4) until the pre-set time window is reached;

[0032] (6) In FAU, the incremental statistics in SSU are aggregated to obtain single-stream features and multi-stream features;

[0033] (7) Repeat steps (1) to (6) to extract features from the traffic data in real time.

[0034] The present invention also provides a system for implementing the method.

[0035] The present invention also provides an application of the method in the field of network security technology.

[0036] The present invention also provides an application of the system in the field of network security technology.

[0037] (III) Beneficial Effects

[0038] This invention addresses the technical challenge of real-time extraction of multi-dimensional traffic features in high-speed network environments for intelligent network threat detection. Combining damped incremental statistics and sketching techniques, it proposes a Feature Incremental Statistical Sketching (ISS) architecture. Based on this architecture, a real-time traffic feature extraction method for high-speed networks is presented, enabling real-time extraction of multi-dimensional traffic features under high-load network conditions. This extraction method possesses high-speed traffic processing capabilities and real-time traffic feature calculation capabilities, allowing for real-time extraction of multi-dimensional traffic features in high-speed network environments. Attached Figure Description

[0039] Figure 1 This is a flowchart of the present invention;

[0040] Figure 2 This is a schematic diagram of an address mapping unit;

[0041] Figure 3 This is a schematic diagram of the sketch storage unit;

[0042] Figure 4 This is a schematic diagram of vertical aggregation within the sketch;

[0043] Figure 5 This is a schematic diagram of vertical aggregation and horizontal aggregation between sketches. Detailed Implementation

[0044] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0045] To address the problem of real-time extraction of multidimensional traffic features in high-speed networks, this invention proposes an Incremental Statistics Sketch (ISS) architecture that combines damped incremental statistics and sketch data structures, and based on this architecture, proposes a method for real-time extraction of traffic features in high-speed networks.

[0046] This invention proposes a real-time method for extracting multi-dimensional features of high-speed network traffic. The scheme comprises two parts: 1) ISS architecture construction method: Combining the damped incremental statistical method with a sketch data structure, the flow ID can be obtained through the Packet Parsing Unit (PPU), the storage address corresponding to the flow ID in the Sketch Storage Unit (SSU) can be calculated through the Address Mapping Unit (AMU), and the incremental statistical value can be stored. The data in the Sketch Storage Unit (SSU) can be aggregated through the Features Aggregate Unit (FAU) to finally obtain multi-dimensional traffic features; 2) Real-time traffic feature extraction method based on the ISS architecture: Based on the ISS architecture, real-time extraction of multi-dimensional traffic features of high-speed networks is achieved.

[0047] The following section will provide a detailed introduction to each part of the plan.

[0048] (I) ISS Architecture Construction

[0049] This section primarily presents the ISS architecture. This architecture mainly consists of four parts: the Packet Parsing Unit (PPU), the Address Mapping Unit (AMU), the Sketch Storage Unit (SSU), and the Feature Aggregate Unit (FAU). Each component is described below.

[0050] (1) PPU:

[0051] The PPU parses traffic data packets, extracting the corresponding flow ID information and protocol field information for each input data packet.

[0052] A flow ID is a 5-tuple of information (source IP address, destination IP address, source port number, destination port number, and protocol type), which can be quantified in some way and used as input to a hash function. For example, the protocol type can be numerically represented and concatenated with the source IP address, destination IP address, source port number, and destination port number to obtain the flow ID.

[0053] Packet parsing functionality can be implemented using underlying libraries such as PcapPlusPlus and tshark.

[0054] (2) AMU:

[0055] The AMU consists of multiple sets of hash functions used to represent the mapping from the stream ID to the Sketch Memory Unit (SSU) storage address;

[0056] like Figure 2 As shown, the number of hash functions in AMU can be represented as N×M, where N is the number of hash function groups and M is the number of hash functions in each group;

[0057] Each set of hash functions consists of M distinct hash functions from the same family of hash functions. Optionally, the family of hash functions can use k-global hash functions, represented as follows:

[0058]

[0059] Where r represents a prime number, a i and b i A is an integer randomly selected from [0, r-1]. i ≠0, L is the length of the hash table.

[0060] In AMU, the number of hash function groups N is the number of incremental statistics, and M is used to control the upper bound of the error of incremental statistics; incremental statistics refer to the intermediate statistics used in incremental statistics to calculate the final statistics.

[0061] (3) SSU:

[0062] SSU consists of multiple hash tables used to store incremental statistics;

[0063] like Figure 3 As shown, multiple hash tables in SSU can be represented as a three-dimensional matrix of size N×M×L, where N is the number of hash tables, M is the number of hash tables in each group, and L is the length of each hash table.

[0064] The hash address of each hash table in SSU is calculated using the hash function of the corresponding group in AMU;

[0065] A set of hash tables in SSU and the corresponding hash functions in AMU form a sketch of shape M×L, with N sketches representing different incremental statistics.

[0066] (4) FAU:

[0067] FAU is implemented by a set of aggregation functions, which are used to aggregate the feature data (incremental statistics) in SSU to obtain the final feature values;

[0068] Aggregation functions include three types: vertical aggregation within a sketch, vertical aggregation between sketches, and horizontal aggregation between sketches.

[0069] like Figure 4As shown, vertical aggregation within a sketch refers to the process of calculating the estimated values ​​of incremental statistics corresponding to all flow IDs for each sketch, forming an array of incremental statistics of length L.

[0070] like Figure 5 As shown, vertical aggregation between sketches refers to the process of combining different incremental statistics of the same flow ID to obtain the final single flow feature after vertical aggregation of all sketches. Single flow feature refers to the flow feature calculated from all flow data with the same flow ID.

[0071] like Figure 5 As shown, horizontal aggregation between sketches refers to the process of calculating the same incremental statistics and flow characteristics of different flow IDs by combining the array of incremental statistics and flow characteristics obtained by vertical aggregation within all sketches and vertical aggregation between sketches; multi-flow characteristics refer to flow characteristics other than single-flow characteristics calculated from all flow data.

[0072] (II) Real-time extraction of traffic features based on ISS architecture

[0073] This section primarily utilizes the ISS architecture to achieve real-time extraction of multi-dimensional traffic characteristics under high-speed network conditions. It mainly includes the following steps:

[0074] (1) Obtain a data packet from the raw data stream;

[0075] (2) Parse the data packets in the PPU to obtain the flow ID and information of each protocol field;

[0076] (3) Obtain the N×M hash addresses corresponding to the stream ID in the AMU;

[0077] (4) Based on the hash address obtained in (3), store the protocol field information of the data packet into the corresponding sketch in SSU and update the incremental statistics;

[0078] (5) Repeat steps (1) to (4) until the pre-set time window is reached;

[0079] (6) In FAU, the incremental statistics in SSU are aggregated to obtain single-stream features and multi-stream features;

[0080] (7) Repeat the above steps to extract features from the traffic data in real time.

[0081] In this embodiment, the traffic features to be extracted are the mean, variance, and standard deviation of the packet sizes. Let P = {P1, P2, P3, ...} represent the original data stream. In incremental statistics, the incremental statistical values ​​required to calculate the above three features can be represented as IS = {Num, LS, SS}, where Num is the current number of packets, LS represents the linear sum of packet sizes, and SS represents the sum of squares of packet sizes. The traffic features can then be calculated as follows: mean variance Standard deviation Therefore, in the ISS architecture of this embodiment, N=3. Let FS={FS1, FS2, FS3} represent the three sketches in the SSU, and all values ​​in them are initialized to 0. Let j = 1, 2, 3, k = 1, 2, ..., M, and let i = 0. The steps for real-time traffic feature extraction of P in this invention are as follows:

[0082] (1) i←i+1, obtain data packet P from P. i The stream ID is obtained by parsing in the PPU. i The size of the data packet is x i ;

[0083] (2) Calculate ID in AMU i The corresponding hash address can be represented as Where j = 1, 2, 3, k = 1, 2, ..., M;

[0084] (3) Update the incremental statistics for each sketch using the following formula:

[0085]

[0086]

[0087]

[0088] FS1 is used to store Num, FS2 is used to store LS, FS3 is used to store SS, and k = 1, 2, ..., M.

[0089] (4) Repeat steps (1) to (3) until the preset time window is reached;

[0090] (5) Aggregate the incremental statistics in FAU to obtain the corresponding traffic characteristics:

[0091] (5a) The estimated values ​​of the incremental statistics obtained by vertical aggregation within the sketch are shown below.

[0092]

[0093]

[0094]

[0095] Among them, ID x Represents a stream ID, These represent the estimated values ​​of the number of packets, the linear sum of packet sizes, and the sum of squares of packet sizes corresponding to the flow ID, respectively.

[0096] (5b) The single-flow characteristics obtained by vertical aggregation between sketches are shown below.

[0097]

[0098]

[0099]

[0100] (5c) The multi-flow features obtained by lateral aggregation between sketches are shown below.

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107] Among them, Num total LS represents the total number of packets. total SS total μ total , σ total These represent the linear sum, sum of squares, mean, variance, and standard deviation of all data packet sizes, respectively.

[0108] (6) Repeat the above steps to achieve real-time extraction of traffic characteristics.

[0109] For ease of description, the above steps are expressed using an incremental statistical method. To ensure the timeliness of extracting flow characteristics, this invention can also use a damped incremental statistical method to calculate flow characteristics. The difference lies in introducing a damping factor when updating the incremental statistical values ​​Num, LS, and SS. Optionally, the damping factor can be expressed as d. λ =2 -λt , where λ>0 is the discount rate, and t represents the data packet arrival time interval.

[0110] Similarly, only 3-dimensional traffic features were extracted in this embodiment. The extraction method for more dimensional traffic features is similar to that described above.

[0111] As can be seen, the technical solution of this invention proposes an ISS architecture and designs a real-time traffic feature extraction method based on the ISS architecture. The core of this invention is the proposal of an ISS architecture for real-time extraction of multi-dimensional traffic features from high-speed networks. This method has the following advantages:

[0112] (1) Low demand for computing resources

[0113] The ISS architecture is based on a sketch data structure and uses multiple hash tables to store statistical feature values. Since the access time of the hash tables depends only on the hash function calculation time and is independent of the amount of data stored; furthermore, traffic features are calculated incrementally, and process data is stored within the hash tables, the overall time complexity is relatively low. Therefore, this technical solution is suitable for real-time extraction of traffic features under high-speed networks.

[0114] (2) The demand for storage resources is relatively fixed.

[0115] The space complexity of the ISS architecture is O(N×M×L), where N, M, and L are predetermined and do not change with network load. Furthermore, the value of N×M×L is much smaller than the traffic volume of a high-speed network.

[0116] (3) It can guarantee the accuracy of feature extraction.

[0117] The sketch data structure ensures that the error in incremental statistics is controllable when the value of N×M×L is much smaller than the traffic volume of the high-speed network. At this point, the error mainly originates from hash collisions in the AMU. Taking the k-global hash function as an example, the probability that two different traffic flows with different IDs are mapped to the same hash address is... In addition, hash collision problems can be solved by methods such as open addressing, chaining, and rehashing, which can further reduce the error of feature extraction.

[0118] (4) Inherent parallelism

[0119] Parallelization is a preferred method for improving processing speed. The ISS architecture is essentially composed of multiple hash tables, controlled by multiple independently working hash functions, and has inherent parallelism, mainly reflected in the following two aspects:

[0120] (4a) Parallel Network Traffic Feature Extraction: Due to the lagging development speed of CPU performance compared to optical communication, a single processing node cannot handle the ultra-large capacity traffic data carried in optical fibers. Therefore, multiple processing nodes are needed to jointly undertake the traffic processing task. At this time, it is necessary to extract network traffic features through multiple data streams. For this purpose, various parallel schemes can be proposed based on the ISS architecture. For example, for each data stream, an independent address mapping unit and a sketch storage unit are used to store the features to be counted. In the final aggregation process, it is only necessary to perform intra-sket aggregation of all sketch storage units. When aggregating between sketches, the values ​​corresponding to the same flow ID in different sketch storage units are additionally aggregated to obtain all traffic features.

[0121] (4b) Parallel processing of feature extraction: When extracting multidimensional traffic features under high load network, since the sketch data structure is composed of multiple hash tables, its structure itself has natural parallelism. Therefore, each data stream of multiple processing nodes can be stored in parallel, which can greatly reduce the time overhead of data storage.

[0122] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for real-time extraction of multi-dimensional features of high-speed network traffic, characterized in that, Includes the following steps: Step 1: Design the Feature Increment Statistical Sketch (ISS) by combining the damping increment statistics and the sketch data structure; Step 2: Real-time extraction of traffic features based on ISS; In step one, the ISS is designed to consist of the following four parts: packet parsing unit PPU, address mapping unit AMU, sketch storage unit SSU, and feature aggregation unit FAU; The PPU is used to parse traffic data packets. For each input data packet, it parses out the corresponding flow ID information and the information of each protocol field. The flow ID refers to the five-tuple information: source IP address, destination IP address, source port number, destination port number, and protocol type. The five-tuple information can be quantized and used as the input of a hash function. The AMU consists of multiple sets of hash functions used to represent the mapping from stream IDs to Sketch Unit (SSU) storage addresses; the number of hash functions in the AMU is expressed as... ,in N The number of groups for the hash function. M The number of hash functions in each group; each group of hash functions consists of elements from the same family of hash functions. M It consists of several different hash functions; the number of hash function groups in AMU N The number of incremental statistics. M The upper bound of error used to control incremental statistics; incremental statistics refer to the intermediate statistical values ​​used in incremental statistics to calculate the final statistical value; The SSU consists of multiple hash tables used to store incremental statistics; the multiple hash tables in the SSU are represented as a set of hash tables of size . N M L A three-dimensional matrix, where N The number of groups in the hash table. M The number of hash tables in each group. L The length of each hash table; the hash address of each hash table in SSU is calculated using the hash function of the corresponding group in AMU; A set of hash tables in SSU and the corresponding hash functions in AMU form a shape of... The sketch, N Each sketch represents a different incremental statistical value; FAU is implemented by a set of aggregation functions, which are used to aggregate the feature data in SSU, i.e., the incremental statistics, to obtain the final feature values. The aggregation functions include three types: vertical aggregation within a sketch, vertical aggregation between sketches, and horizontal aggregation between sketches. Vertical aggregation within a sketch refers to, for each sketch, calculating estimates of the incremental statistics corresponding to all flow IDs, forming a length of... L The process of creating an array of incremental statistical values; Vertical aggregation between sketches refers to the process of combining different incremental statistics of the same flow ID to obtain the final single flow feature after vertical aggregation of all sketches. Single flow feature refers to the flow feature calculated from all flow data with the same flow ID. Horizontal aggregation between sketches refers to the process of calculating the same incremental statistics and flow characteristics of different flow IDs by combining the array of incremental statistics obtained after vertical aggregation within all sketches and the single flow flow characteristics obtained by vertical aggregation between sketches; multi-flow characteristics refer to flow characteristics other than single flow characteristics calculated from all flow data.

2. The method as described in claim 1, characterized in that, In PPU, the protocol type can be numerically represented and concatenated with information such as source IP address, destination IP address, source port number, and destination port number to obtain the flow ID.

3. The method as described in claim 1, characterized in that, The PPU's packet parsing function can be implemented through one of the underlying libraries, PcapPlusPlus or tshark.

4. The method as described in claim 1, characterized in that, Hash function family k - Global hash function, represented as in, To represent a prime number, and From A randomly selected integer. , This represents the length of the hash table.

5. The method as described in claim 1, characterized in that, Step two specifically includes the following steps: (1) Obtain a data packet from the raw data stream; (2) Parse the data packets in the PPU to obtain the flow ID and information of each protocol field; (3) Obtain the stream ID corresponding to the stream ID in the AMU. A hash address; (4) Based on the hash address obtained in step (3), store the protocol field information of the data packet into the corresponding sketch in SSU and update the incremental statistics. (5) Repeat steps (1) to (4) until the pre-set time window is reached; (6) In FAU, the incremental statistics in SSU are aggregated to obtain single-stream features and multi-stream features; (7) Repeat steps (1) to (6) to extract features from the traffic data in real time.

Images