Flow feature extraction method and interface based on programmable network

Abstract

The invention discloses a traffic feature extraction method and interface based on a programmable network, and relates to the technical field of traffic feature extraction, and the method comprises the specific steps: carrying out the abstract processing of a data package, and obtaining a data package tuple; grouping the data packet tuples according to specified granularity to obtain group tuples; screening the data packet tuple or the group tuple based on a preset screening condition to obtain a screened tuple; and mapping the screening tuple to generate a mapping tuple, performing aggregation calculation on the mapping tuple to obtain a feature sequence, and converting the feature sequence into a feature vector. According to the method, the implementation details of bottom hardware can be simplified, the complexity is reduced, and meanwhile, a universal feature extraction method which is closely combined with traffic analysis application characteristics has important significance in improving the overall performance of a network traffic analysis system and promoting the development of a network security technology.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic feature extraction, and more specifically to a traffic feature extraction method and interface based on a programmable network. Background Art

[0002] With the rapid development of network communication technology, network traffic analysis has become an important means to ensure network security and identify malicious behaviors. Traditional network traffic analysis methods mainly rely on the analysis of payloads. However, with the widespread application of encryption technology, this method has gradually become ineffective. In order to meet this challenge, machine learning (ML), especially deep learning technology, has been introduced into network traffic analysis to improve the accuracy and efficiency of analysis.

[0003] Traffic analysis technology based on machine learning usually includes two key components: feature extractor and behavior detector. The feature extractor is responsible for extracting necessary traffic features from raw network traffic and converting them into feature vectors, while the behavior detector uses machine learning algorithms to analyze these feature vectors to identify various network malicious activities, such as botnets, covert channels, and network intrusions.

[0004] However, although behavior detectors have made significant progress in utilizing hardware acceleration such as GPUs and TPUs, feature extractors still face a large bottleneck. The current mainstream feature extractors mainly use software-based methods to replicate network traffic through port mirroring and use a large number of servers for storage and feature extraction. This method not only has high communication, storage and computing overhead, but is also inadequate when dealing with high-speed networks of hundreds of Gbps.

[0005] To overcome this challenge, emerging programmable network hardware in the network field, such as programmable switches and smart network cards, provide new possibilities for efficient and flexible feature extraction. These hardware not only support extremely high packet processing performance, but also provide programming interfaces that allow users to implement customized packet processing logic. Some research works have attempted to use these hardware to accelerate feature extractors and implement feature extraction or even the entire traffic analysis task in the data plane, greatly reducing the demand for additional bandwidth and servers.

[0006] However, these works still have some limitations. First, most of them only target specific traffic analysis applications, lack versatility and flexibility, and cannot support a variety of traffic analysis processes. Second, directly implementing feature extractors based on programmable network hardware requires a deep understanding of hardware programming models and resource limitations, which increases the complexity for users.

[0007] The development of programmable switches and the P4 language has further promoted the advancement of network programmability. Programmable switches combine the power consumption and price advantages of traditional switches while providing programming capabilities that allow users to execute customized packet processing logic at line speed. As a domain-specific programming language adapted to the hardware architecture of programmable switches, the P4 language provides programmers with the ability to write and run P4 programs to implement various network applications. However, writing P4 programs requires a detailed understanding of product attributes and hardware architecture, which poses challenges to the implementation and migration of applications such as network traffic analysis.

[0008] As another new type of programmable network card, the Smart NIC is equipped with a powerful processor, memory and dedicated hardware, which can perform network computing, storage acceleration and security protection at line speed. By writing Micro-C programs, the Smart NIC can realize streaming processing of data packets. However, this also requires users to have high professional skills and a detailed understanding of the hardware resources and programming model of the network card.

[0009] Therefore, how to develop a method that can hide the underlying hardware implementation details and complexity and provide users with a simple and general feature extraction method in combination with the traffic analysis application feature extraction process is an urgent problem that technicians in this field need to solve. Summary of the invention

[0010] In view of this, the present invention provides a traffic feature extraction method and interface based on a programmable network, which overcomes the above-mentioned defects.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] A method for extracting traffic features based on a programmable network, the specific steps are:

[0013] Perform abstract processing on the data packet to obtain the data packet tuple;

[0014] Grouping the data packet tuples according to a specified granularity to obtain group tuples;

[0015] Filter the data packet tuple or the group tuple based on a preset filtering condition to obtain a filtered tuple;

[0016] The screening tuple is mapped to generate a mapping tuple, and the mapping tuple is aggregated to obtain a feature sequence, and the feature sequence is converted into a feature vector.

[0017] The step of acquiring the data packet tuple is to abstract the data packet using a key-value pair to obtain the data packet tuple.

[0018] Optionally, the key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as the key and the value of the data packet as the value; the other category uses the metadata of the data packet as the key and the value generated according to the metadata of the data packet as the value.

[0019] Optionally, the group tuple includes two key-value pairs, the first key-value pair is used to record the granularity value of the group of data packets; the second key-value pair is used to represent a list of the data packet tuples included in the group.

[0020] Optionally, in the process of acquiring the group tuples, multi-layer grouping is performed according to different specified granularities.

[0021] Optionally, the steps of obtaining the mapping tuple are:

[0022] Generate a mapping group from each group in the group tuple according to a mapping function; and construct the mapping tuple according to the mapping group and the screening tuple pair.

[0023] Optionally, the step of acquiring the feature sequence is:

[0024] Determine the reduction function according to the required target;

[0025] Calculate the target feature using the reduction function based on the mapping tuple;

[0026] The target feature is added to the mapping tuple, and processed using a comprehensive function to obtain the feature sequence.

[0027] Optionally, the step of acquiring the feature vector is:

[0028] The feature sequence is converted into the feature vector according to preset conversion parameters.

[0029] Optionally, the conversion parameter is the generation frequency of the feature vector.

[0030] A traffic feature extraction system based on a programmable network, comprising:

[0031] The feature extraction module is used to perform abstract processing on the data packet to obtain the data packet tuple;

[0032] A grouping module, used for grouping the data packet tuples according to a specified granularity to obtain group tuples;

[0033] A screening module, used for screening the data packet tuple or the group tuple based on a preset screening condition to obtain a screening tuple;

[0034] The feature processing module is used to map the screening tuple to generate a mapping tuple, perform aggregation calculation on the mapping tuple to obtain a feature sequence, and convert the feature sequence into a feature vector.

[0035] It can be seen from the above technical solutions that the present invention discloses a traffic feature extraction method and interface based on a programmable network, which has the following beneficial effects compared with the prior art:

[0036] Efficiency: The present invention directly extracts traffic features on programmable network devices, avoiding the delay of transmitting data packets to the server before processing in the traditional method, thereby improving the speed and efficiency of feature extraction;

[0037] Flexibility: The present invention supports grouping and screening of data packets at different granularities, and can adjust feature extraction strategies according to different application scenarios, thus enhancing the adaptability and flexibility of the system;

[0038] Scalability: By utilizing hardware features such as programmable switches, the system's processing power and functions can be easily expanded to meet the growing network traffic demand;

[0039] Reduce resource consumption: By completing feature extraction at the edge of the network, the pressure on the central server is reduced, network bandwidth and server resources are saved, and the overall operating cost is reduced;

[0040] Enhanced security: Since most of the processing is done inside the network equipment, the risk of data leakage is reduced and the security of the system is improved;

[0041] Simplify the development process: An abstraction layer is proposed to hide the specific implementation details of the underlying hardware, making it easier for developers to write feature extraction logic without having to have an in-depth understanding of complex hardware programming models, thus lowering the development threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A schematic diagram of the method provided by the present invention;

[0044] Figure 2 A schematic diagram of the process flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] On the one hand, the present invention discloses a method for extracting traffic characteristics based on a programmable network, such as Figure 1 As shown, the specific steps are:

[0047] Perform abstract processing on the data packet to obtain the data packet tuple;

[0048] Grouping the data packet tuples according to the specified granularity to obtain group tuples;

[0049] Filter the data packet tuple or group tuple based on the preset filtering condition to obtain a filtered tuple;

[0050] The filter tuple is mapped to generate a mapping tuple, and the mapping tuple is aggregated to obtain a feature sequence, and the feature sequence is converted into a feature vector.

[0051] In one embodiment, the step of obtaining the data packet tuple is to abstract the data packet using the key-value pair tuple to obtain the data packet tuple.

[0052] In one embodiment, the key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as the key and the value of the data packet as the value; the other category uses the metadata of the data packet as the key and the value generated according to the metadata of the data packet as the value.

[0053] Furthermore, in step 1, the SuperFE interface uses a key-value pair tuple to abstract the data packet, wherein the key-value pair includes two categories: one category uses the header field of the data packet as the key and the corresponding value in the data packet as the value, such as the source / destination IP address of the data packet; the other category uses the metadata of the data packet as the key, and the corresponding value is filled in by the programmable switch, such as the size of the data packet and the message arrival timestamp. When used, the user is allowed to extend this tuple abstraction to include any field that the programmable switch can parse or obtain. In the programming model of the feature extraction strategy, the input of the strategy is a data packet stream, that is, a series of data packet key-value tuples, which are represented by pktstream in the SuperFE interface, and each function in the SuperFE interface is implemented by operators. In addition to the data packet, the intermediate data transmitted by SuperFE between operators is also represented in the form of a key-value tuple.

[0054] In one embodiment, the group tuple includes two key-value pairs, the first key-value pair is used to record the granularity value of the group of data packets; the second key-value pair is used to represent a list of key-value tuples included in the group.

[0055] Furthermore, since traffic analysis applications usually only care about a few common granularity traffic features, such as the packet rate corresponding to each source IP address or each flow, a packet stream processing model based on grouping is proposed. For the input pktstream of the feature extraction strategy, that is, the key-value tuples corresponding to the packets in the original traffic (i.e., packet tuples), users can divide them into different groups according to the specified granularity g, which can be implemented using the groupby(g) operator in the SuperFE interface. Its output is also a series of key-value tuples, each of which corresponds to a group (referred to as group tuple) consisting of packet tuples with the same granularity value. Each group tuple obtained by grouping contains two key-value pairs: one is the "granularity" key-value pair, which is used to record the granularity value of the group of packets; the other is the "member" key-value pair, whose value is the list of packet tuples contained in the group.

[0056] In one embodiment, during the process of acquiring the tuples, multi-layer grouping may be performed according to different specified granularities.

[0057] Furthermore, similar to the grouping of data packet tuples, the same method can be used to perform further grouping operations on group tuples at a coarser granularity. By providing such a multi-layer grouping function, it is possible to support the feature extraction requirements of more complex traffic analysis applications, such as extracting features such as "the number of TCP flows established by each IP address". Of course, the operator groupby(g) can also be easily extended to support more grouping granularities g. Through the above-mentioned group-based streaming processing model, users can intuitively describe the feature extraction logic of their traffic analysis applications.

[0058] Furthermore, the step of obtaining the filter tuple is: select the packet tuple or group tuple that satisfies the assertion p from the input key-value tuple, and transmit it to the next step, to help the user filter out the traffic that the traffic analysis application is really interested in, which can be implemented according to the filter(p) operator in the SuperFE interface.

[0059] In one embodiment, the steps of obtaining the mapping tuple are:

[0060] Generate mapping groups from each group in the group tuple according to the mapping function; construct a mapping tuple according to the mapping group and the filter tuple pair.

[0061] Furthermore, due to the unique processing model of the SuperFE interface based on packets, the subsequent processing scope is limited to one group. Based on the mapping function, a mapping operation is performed on each (data packet / group) tuple in the "member" list of the input group tuple, that is, the mapping function mf is applied to the source key-value pair s of each member tuple and the result is saved in the new target key-value pair d. It is mainly responsible for extracting some intermediate statistical data from the original traffic through preprocessing for subsequent feature calculations, which can be implemented according to the map(d,s,mf) operator in the SuperFE interface.

[0062] In one embodiment, the steps of acquiring the feature sequence are:

[0063] Determine the reduction function according to the required target;

[0064] Calculate the target feature using the reduction function based on the mapping tuple;

[0065] The target features are added to the mapping tuple and processed using the comprehensive function to obtain the feature sequence.

[0066] Furthermore, all key-value pairs s of each "member" tuple of the input group tuple are aggregated, and a set of reduction functions are applied to these key-value pairs to calculate the required features, which are added as new key-value pairs to each input group tuple by the SuperFE interface, which can be implemented according to the reduce(s,[rf]) operator in the SuperFE interface.

[0067] Furthermore, you can also call synthesize(sf) in the SuperFE interface to further process the feature sequence using the synthesis function sf.

[0068] In one embodiment, the steps of acquiring the feature vector are:

[0069] The feature sequence is converted into a feature vector according to the preset conversion parameters.

[0070] Furthermore, according to the collect(u) operator in the SuperFE interface, SuperFE can be instructed how to generate the final feature vector based on the feature tuple calculated previously.

[0071] In one embodiment, the conversion parameter is the generation frequency of the feature vector.

[0072] Furthermore, the conversion parameter u indicates the frequency of generating the feature vector, which can be pkt or a certain granularity g, meaning whether the final feature vector is generated packet by packet or group by group at a certain granularity.

[0073] In one embodiment, the mapping functions, reduction functions and synthesis functions supported by the SuperFE interface are shown in Table 1, and these functions can also be extended.

[0074] Table 1 Description of SuperFE interface parameters

[0075]

[0076]

[0077] In this embodiment, a traffic feature extraction interface based on a programmable network is also disclosed. It is a feature extraction interface based on a programmable switch and an intelligent network card, namely, a SuperFE interface, which can hide the underlying hardware implementation details and hardware complexity, and provide users with a set of simple and universal feature extraction interfaces in combination with the process of feature extraction of traffic analysis applications, including:

[0078] The feature extraction module is used to perform abstract processing on the data packet to obtain the data packet tuple;

[0079] A grouping module, used for grouping data packet tuples according to a specified granularity to obtain group tuples;

[0080] A screening module, used for screening the data packet tuple or group tuple based on a preset screening condition to obtain a screening tuple;

[0081] The feature processing module is used to map the filter tuple to generate a mapping tuple, perform aggregation calculation on the mapping tuple to obtain a feature sequence, and convert the feature sequence into a feature vector.

[0082] Furthermore, although different traffic analysis applications require different protocol semantics and traffic features and are suitable for different application scenarios, there is a common feature extraction process under these different applications - selecting network traffic, grouping network traffic, mapping intermediate statistics, and generating the final feature vector. This universality provides an opportunity for the abstract expression of the feature extraction process at a high level. Based on the programmable capabilities provided by programmable switches and smart network cards, and inspired by Spark-style stream processing operators, the SuperFE interface implements the encapsulation of network traffic feature extraction operations at a higher level. Its main operators are shown in Table 2.

[0083] Table 2 Main operators of SuperFE interface

[0084]

[0085] In one embodiment, Figure 2The flow processing process of calculating the average, maximum and minimum interval time of TCP packets sent by each IP host is presented. First, the filter operator is used to select TCP packet tuples from the network traffic. Then, the groupby operator is used to group the data according to the source IP. Then, the map operator is used to map out the time interval ipt between packets. Then, the reduce operator is used to obtain the average, maximum and minimum values ​​of ipt. Finally, the collect operator is used to convert the features obtained by reduce into feature vectors. Various flow processing processes that meet user needs can be implemented by combining various operators and customizing parameters.

[0086] In one embodiment, the present invention requires the use of Python and other codes to implement a policy engine, which is a simple translator that can configure the P4 program on the switch by analyzing the user-defined feature extraction strategy, extracting the groupby and filter operators, and extracting the remaining operators (map, reduce, synthesize, collect) to configure the Micro-C program on the smart network card.

[0087] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0088] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting traffic features based on programmable networks, characterized in that: The specific steps are: Perform abstract processing on the data packet to obtain the data packet tuple; Grouping the data packet tuples according to a specified granularity to obtain group tuples; Filter the data packet tuple or the group tuple based on a preset filtering condition to obtain a filtered tuple; The screening tuple is mapped to generate a mapping tuple, and the mapping tuple is aggregated to obtain a feature sequence, and the feature sequence is converted into a feature vector.

2. According to a method for extracting traffic characteristics based on a programmable network according to claim 1, it is characterized in that: The step of acquiring the data packet tuple is to abstract the data packet using a key-value pair to obtain the data packet tuple.

3. According to claim 2, a method for extracting traffic characteristics based on a programmable network is characterized in that: The key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as the key and the value of the data packet as the value; the other category uses the metadata of the data packet as the key and the value generated according to the metadata of the data packet as the value.

4. The method for extracting traffic characteristics based on a programmable network according to claim 2 is characterized in that: The group tuple includes two key-value pairs, the first key-value pair is used to record the granularity value of the group of data packets; the second key-value pair is used to represent the list of the data packet tuples included in the group.

5. The method for extracting traffic characteristics based on a programmable network according to claim 1 is characterized in that: In the process of acquiring the group tuples, multi-layer grouping is performed according to different specified granularities.

6. The method for extracting traffic characteristics based on a programmable network according to claim 1 is characterized in that: The steps for obtaining the mapping tuple are: Generate a mapping group from each group in the group tuple according to a mapping function; and construct the mapping tuple according to the mapping group and the screening tuple pair.

7. The method for extracting traffic characteristics based on a programmable network according to claim 1 is characterized in that: The steps of obtaining the feature sequence are: Determine the reduction function according to the required target; Calculate the target feature using the reduction function based on the mapping tuple; The target feature is added to the mapping tuple, and processed using a comprehensive function to obtain the feature sequence.

8. The method for extracting traffic characteristics based on a programmable network according to claim 1 is characterized in that: The steps of obtaining the feature vector are: The feature sequence is converted into the feature vector according to preset conversion parameters.

9. The method for extracting traffic characteristics based on a programmable network according to claim 8, characterized in that: The conversion parameter is the generation frequency of the feature vector.

10. A traffic feature extraction system based on a programmable network, characterized in that: include: The feature extraction module is used to perform abstract processing on the data packet to obtain the data packet tuple; A grouping module, used for grouping the data packet tuples according to a specified granularity to obtain group tuples; A screening module, used for screening the data packet tuple or the group tuple based on a preset screening condition to obtain a screening tuple; The feature processing module is used to map the screening tuple to generate a mapping tuple, perform aggregation calculation on the mapping tuple to obtain a feature sequence, and convert the feature sequence into a feature vector.

Images