A traffic feature extraction method and interface based on programmable network
By abstractly processing, grouping, and mapping data packets on a programmable network to generate feature sequences, the high overhead and complexity problems of existing feature extractors are solved, efficient, flexible, and secure feature extraction is achieved, and the development process is simplified.
Patent Information
- Application Number
- CN202411880764.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-09-12
- Estimated Expiration
- 2044-12-19
AI Technical Summary
Existing software-based feature extractors have high communication, storage, and computational overheads in network traffic analysis, and users need to have an in-depth understanding of the hardware programming model. They lack versatility and flexibility and are difficult to adapt to various traffic analysis processes.
A traffic feature extraction method based on programmable networks is adopted to generate feature sequences by abstractly processing, grouping, filtering and mapping data packets, and finally converting them into feature vectors. The SuperFE interface is provided to hide hardware details and simplify the development process.
It improves the speed and efficiency of feature extraction, enhances the adaptability and security of the system, reduces resource consumption and operating costs, and simplifies development difficulty.
Smart Images
Figure CN119728232B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of traffic feature extraction, and more particularly to a traffic feature extraction method and interface based on a programmable network. Background Art
[0002] With the rapid development of network communication technologies, network traffic analysis has become a crucial tool for ensuring network security and identifying malicious behavior. Traditional network traffic analysis methods primarily rely on parsing payloads, but with the widespread adoption of encryption technologies, this approach is becoming increasingly ineffective. To address this challenge, machine learning (ML), particularly deep learning, has been introduced into network traffic analysis to improve both accuracy and efficiency.
[0003] Machine learning-based traffic analysis techniques typically consist of two key components: a feature extractor and a behavior detector. The feature extractor extracts essential traffic features from raw network traffic and converts them into feature vectors. The behavior detector, on the other hand, uses machine learning algorithms to analyze these feature vectors to identify various malicious network activities, such as botnets, covert channels, and network intrusions.
[0004] However, despite significant progress in leveraging hardware acceleration for behavior detectors like GPUs and TPUs, feature extraction still faces significant bottlenecks. Current mainstream feature extraction methods primarily rely on software-based approaches, replicating network traffic through port mirroring and utilizing a large number of servers for storage and feature extraction. This approach not only incurs high communication, storage, and computational overhead, but also proves inadequate when dealing with high-speed networks running at hundreds of Gbps.
[0005] To overcome this challenge, emerging programmable network hardware, such as programmable switches and SmartNICs, offers new possibilities for efficient and flexible feature extraction. This hardware not only supports extremely high packet processing performance but also provides programming interfaces that allow users to implement customized packet processing logic. Several research efforts have attempted to leverage this hardware to accelerate feature extraction and implement feature extraction, or even entire traffic analysis tasks, in the data plane, significantly reducing the need for additional bandwidth and servers.
[0006] However, these works still have some limitations. First, most of them only target specific traffic analysis applications and lack versatility and flexibility to support a variety of traffic analysis processes. Second, directly implementing feature extractors on programmable network hardware requires a deep understanding of the hardware programming model and resource constraints, which increases user complexity.
[0007] The development of programmable switches and the P4 programming language has further advanced network programmability. Programmable switches combine the power consumption and price advantages of traditional switches with programmability, allowing users to execute customized packet processing logic at line speed. As a domain-specific programming language adapted to the hardware architecture of programmable switches, P4 provides programmers with the ability to write and run P4 programs to implement a variety of network applications. However, writing P4 programs requires a detailed understanding of product attributes and hardware architecture, which poses challenges to the implementation and migration of applications such as network traffic analysis.
[0008] SmartNICs, another new type of programmable network interface card (NIC), are equipped with powerful processors, memory, and specialized hardware, enabling line-speed network computing, storage acceleration, and security protection. By programming in Micro-C, SmartNICs can stream data packets. However, this requires a high level of technical expertise and a detailed understanding of the NIC's hardware resources and programming model.
[0009] Therefore, how to develop a feature extraction method that can hide the underlying hardware implementation details and complexity and provide users with a simple and universal feature extraction process in combination with the traffic analysis application feature extraction process is an urgent problem that technicians in this field need to solve. Summary of the Invention
[0010] In view of this, the present invention provides a traffic feature extraction method and interface based on a programmable network, which overcomes the above-mentioned defects.
[0011] In order to achieve the above object, the present invention adopts the following technical solutions:
[0012] A method for extracting traffic characteristics based on programmable networks, comprising the following steps:
[0013] Perform abstract processing on the data packet to obtain the data packet tuple;
[0014] Grouping the data packet tuples according to a specified granularity to obtain group tuples;
[0015] Filtering the data packet tuple or the group tuple based on a preset filtering condition to obtain a filtered tuple;
[0016] The screening tuple is mapped to generate a mapping tuple, and the mapping tuple is aggregated to obtain a feature sequence, and the feature sequence is converted into a feature vector.
[0017] The step of obtaining the data packet tuple is to abstract the data packet using a key-value pair to obtain the data packet tuple.
[0018] Optionally, the key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as the key and the value of the data packet as the value; the other category uses the metadata of the data packet as the key and the value generated according to the metadata of the data packet as the value.
[0019] Optionally, the group tuple includes two key-value pairs, the first key-value pair is used to record the granularity value of the group of data packets; and the second key-value pair is used to represent a list of the data packet tuples included in the group.
[0020] Optionally, during the tuple acquisition process, multi-layer grouping is performed according to different specified granularities.
[0021] Optionally, the steps of obtaining the mapping tuple are:
[0022] Each group in the group tuple is used to generate a mapping group according to a mapping function; and the mapping tuple is constructed according to the mapping group and the screening tuple pair.
[0023] Optionally, the step of acquiring the feature sequence is:
[0024] Determine the reduction function according to the required target;
[0025] Calculating a target feature using the reduction function based on the mapping tuple;
[0026] The target feature is added to the mapping tuple, and processed using a comprehensive function to obtain the feature sequence.
[0027] Optionally, the step of obtaining the feature vector is:
[0028] The feature sequence is converted into the feature vector according to preset conversion parameters.
[0029] Optionally, the conversion parameter is the generation frequency of the feature vector.
[0030] A traffic feature extraction system based on a programmable network, comprising:
[0031] Feature extraction module, used to abstract data packets and obtain data packet tuples;
[0032] A grouping module, configured to group the data packet tuples according to a specified granularity to obtain group tuples;
[0033] A screening module, configured to screen the data packet tuple or the group tuple based on a preset screening condition to obtain a screening tuple;
[0034] The feature processing module is used to map the screening tuple to generate a mapping tuple, perform aggregation calculation on the mapping tuple to obtain a feature sequence, and convert the feature sequence into a feature vector.
[0035] Through the above technical solutions, it can be seen that the present invention discloses a traffic feature extraction method and interface based on a programmable network, which has the following beneficial effects compared with the existing technology:
[0036] Efficiency: By directly extracting traffic features on programmable network devices, this method avoids the delay of traditional methods that require data packets to be transmitted to the server before processing, thereby improving the speed and efficiency of feature extraction.
[0037] Flexibility: The present invention supports grouping and filtering of data packets at different granularities and can adjust feature extraction strategies according to different application scenarios, thus enhancing the adaptability and flexibility of the system.
[0038] Scalability: Utilizing hardware features such as programmable switches, the system's processing power and functionality can be easily expanded to meet growing network traffic demands.
[0039] Reduced resource consumption: By completing feature extraction at the network edge, the pressure on the central server is reduced, network bandwidth and server resources are saved, and overall operating costs are reduced;
[0040] Enhanced security: Since most of the processing is completed within the network equipment, the risk of data leakage is reduced and the security of the system is improved;
[0041] Simplified development process: An abstraction layer is proposed to hide the specific implementation details of the underlying hardware, making it easier for developers to write feature extraction logic without having to deeply understand the complex hardware programming model, thus lowering the development threshold. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0043] Figure 1 A schematic flow chart of the method provided by the present invention;
[0044] Figure 2 This is a schematic diagram of the process flow of the embodiment method provided by the present invention. DETAILED DESCRIPTION
[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0046] On the one hand, the present invention discloses a method for extracting traffic characteristics based on a programmable network. Figure 1 As shown, the specific steps are:
[0047] Perform abstract processing on the data packet to obtain the data packet tuple;
[0048] Grouping the data packet tuples according to the specified granularity to obtain group tuples;
[0049] Filter the data packet tuple or group tuple based on the preset filtering condition to obtain a filtered tuple;
[0050] The filter tuple is mapped to generate a mapping tuple, and the mapping tuple is aggregated to obtain a feature sequence, which is then converted into a feature vector.
[0051] In one embodiment, the step of obtaining the data packet tuple is to abstract the data packet using the key-value pair tuple to obtain the data packet tuple.
[0052] In one embodiment, the key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as the key and the value of the data packet as the value; the other category uses the metadata of the data packet as the key and the value generated according to the metadata of the data packet as the value.
[0053] Furthermore, in step 1, the SuperFE interface uses a key-value pair tuple to abstract the data packet, where the key-value pair includes two categories: one category uses the header field of the data packet as the key and the corresponding value in the data packet as the value, such as the source / destination IP address of the data packet; the other category uses the metadata of the data packet as the key, and the corresponding value is filled in by the programmable switch, such as the size of the data packet and the message arrival timestamp. When used, the user is allowed to extend this tuple abstraction to include any field that the programmable switch can parse or obtain. In the programming model of the feature extraction strategy, the input of the strategy is a data packet stream, that is, a series of data packet key-value tuples, which are represented by pktstream in the SuperFE interface. Each function in the SuperFE interface is implemented by operators. In addition to data packets, the intermediate data transmitted by SuperFE between operators is also represented in the form of key-value tuples.
[0054] In one embodiment, the group tuple includes two key-value pairs, the first key-value pair is used to record the granularity value of the group of data packets; the second key-value pair is used to represent a list of key-value tuples included in the group.
[0055] Furthermore, since traffic analysis applications typically only care about a few common granularity traffic features, such as the packet rate per source IP address or per flow, a packet stream processing model based on grouping is proposed. The input to the feature extraction strategy, pktstream, is the key-value tuples corresponding to the packets in the raw traffic (i.e., packet tuples). Users can divide it into different groups according to the specified granularity g, which can be achieved using the groupby(g) operator in the SuperFE interface. The output is also a series of key-value tuples, each corresponding to a group of packet tuples with the same granularity value (referred to as group tuples). Each group tuple obtained through grouping contains two key-value pairs: a "granularity" key-value pair, which records the granularity value of the grouped packets; and a "member" key-value pair, whose value is a list of the packet tuples contained in the group.
[0056] In one embodiment, during the tuple acquisition process, multi-layer grouping may be performed according to different specified granularities.
[0057] Furthermore, similar to the grouping of packet tuples, further grouping operations can be performed on group tuples at a coarser granularity using the same method. By providing this multi-layer grouping capability, more complex feature extraction requirements for traffic analysis applications can be supported, such as extracting features such as "the number of TCP flows established by each IP address." Of course, the operator groupby(g) can also be easily extended to support more granularities g. Through the above-described group-based streaming processing model, users can intuitively describe the feature extraction logic of their traffic analysis applications.
[0058] Furthermore, the step of obtaining the filter tuple is: select the packet tuple or group tuple that satisfies the assertion p from the input key-value tuple, and transmit it to the next step, to help users filter out the traffic that the traffic analysis application is really interested in, which can be implemented according to the filter(p) operator in the SuperFE interface.
[0059] In one embodiment, the steps for obtaining the mapping tuple are:
[0060] Generate mapping groups based on the mapping function for each group in the group tuple; construct a mapping tuple based on the mapping group and the filter tuple pair.
[0061] Furthermore, due to the unique processing model of the SuperFE interface based on groups, the subsequent processing scope is limited to one group. Based on the mapping function, a mapping operation is performed on each (data packet / group) tuple in the "member" list of the input group tuple, that is, the mapping function mf is applied to the source key-value pair s of each member tuple and the result is saved to the new target key-value pair d. It is mainly responsible for extracting some intermediate statistical data from the original traffic through preprocessing for subsequent feature calculations, which can be implemented according to the map(d,s,mf) operator in the SuperFE interface.
[0062] In one embodiment, the steps of obtaining the feature sequence are:
[0063] Determine the reduction function according to the required target;
[0064] Calculate the target feature using the reduction function based on the mapping tuple;
[0065] The target features are added to the mapping tuple and processed using the comprehensive function to obtain the feature sequence.
[0066] Furthermore, all key-value pairs s of each "member" tuple of the input group tuple are aggregated, and a set of reduction functions are applied to these key-value pairs to calculate the required features. These features are added to each input group tuple as new key-value pairs by the SuperFE interface, which can be implemented according to the reduce(s,[rf]) operator in the SuperFE interface.
[0067] Furthermore, you can also call synthesize(sf) in the SuperFE interface to further process the feature sequence using the synthesis function sf.
[0068] In one embodiment, the steps for obtaining the feature vector are:
[0069] The feature sequence is converted into a feature vector according to the preset conversion parameters.
[0070] Furthermore, the collect(u) operator in the SuperFE interface can be used to instruct SuperFE how to generate the final feature vector based on the previously calculated feature tuples.
[0071] In one embodiment, the conversion parameter is the generation frequency of the feature vector.
[0072] Furthermore, the conversion parameter u indicates the frequency of generating feature vectors, which can be pkt or a certain granularity g, meaning whether the final feature vectors are generated packet by packet or group by group at a certain granularity.
[0073] In one embodiment, the mapping functions, reduction functions, and synthesis functions supported by the SuperFE interface are shown in Table 1, and these functions can also be extended.
[0074] Table 1 Description of SuperFE interface parameters
[0075]
[0076]
[0077] This embodiment also discloses a programmable network-based traffic feature extraction interface. This is a feature extraction interface based on programmable switches and smart network cards, namely the SuperFE interface. It can hide the underlying hardware implementation details and hardware complexity, and combines the process of feature extraction for traffic analysis applications to provide users with a set of simple and universal feature extraction interfaces, including:
[0078] Feature extraction module, used to abstract data packets and obtain data packet tuples;
[0079] A grouping module, used to group data packet tuples according to a specified granularity to obtain group tuples;
[0080] A screening module, configured to screen the data packet tuple or group tuple based on a preset screening condition to obtain a screening tuple;
[0081] The feature processing module is used to map the filter tuple to generate a mapping tuple, perform aggregation calculation on the mapping tuple to obtain a feature sequence, and convert the feature sequence into a feature vector.
[0082] Furthermore, while different traffic analysis applications require different protocol semantics and traffic characteristics and are applicable to different application scenarios, these diverse applications share a common feature extraction process: selecting network traffic, grouping it, mapping intermediate statistics, and generating the final feature vector. This universality provides an opportunity for high-level abstraction of the feature extraction process. The SuperFE interface, based on the programmability provided by programmable switches and SmartNICs and inspired by Spark-style stream processing operators, encapsulates network traffic feature extraction operations at a higher level. Its main operators are shown in Table 2.
[0083] Table 2 Main operators of SuperFE interface
[0084]
[0085] In one embodiment, Figure 2This example presents a stream processing procedure for calculating the average, maximum, and minimum intervals between TCP packets sent by each IP host. The filter operator is used to select TCP packet tuples from network traffic. The groupby operator is then used to group the data by source IP address. The map operator is then used to map out the intervals (IPts) between packets. The reduce operator is then used to obtain the average, maximum, and minimum IPts. Finally, the collect operator is used to convert the reduced features into feature vectors. By combining various operators and customizing parameters, various stream processing procedures can be implemented to meet user needs.
[0086] In one embodiment, the present invention requires the use of Python or other codes to implement a policy engine, which is a simple translator that can analyze the user-defined feature extraction strategy, extract the groupby and filter operators to configure the P4 program on the switch, and extract the remaining operators (map, reduce, synthesize, collect) to configure the Micro-C program on the smart network card.
[0087] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0088] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for extracting traffic characteristics based on programmable networks, characterized in that: The specific steps are: Abstracting the data packet to obtain a data packet tuple; the step of obtaining the data packet tuple is to abstract the data packet using a key-value pair by a SuperFE interface to obtain the data packet tuple; the key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as a key and the value of the data packet as a value; the other category uses the metadata of the data packet as a key and a value generated according to the metadata of the data packet as a value; Grouping the data packet tuples according to a specified granularity to obtain group tuples; Filtering the data packet tuple or the group tuple based on a preset filtering condition to obtain a filtered tuple; The screening tuple is mapped to generate a mapping tuple, and the mapping tuple is aggregated to obtain a feature sequence, and the feature sequence is converted into a feature vector.
2. A method for extracting traffic characteristics based on a programmable network according to claim 1, characterized in that: The group tuple includes two key-value pairs, the first key-value pair is used to record the granularity value of the group of data packets; the second key-value pair is used to represent a list of the data packet tuples included in the group.
3. The method for extracting traffic characteristics based on a programmable network according to claim 1, characterized in that: In the process of acquiring the group tuples, multi-layer grouping is performed according to different specified granularities.
4. The method for extracting traffic characteristics based on a programmable network according to claim 1, characterized in that: The steps for obtaining the mapping tuple are: Each group in the group tuple is used to generate a mapping group according to a mapping function; and the mapping tuple is constructed according to the mapping group and the screening tuple pair.
5. The method for extracting traffic characteristics based on a programmable network according to claim 1, characterized in that: The steps for obtaining the feature sequence are: Determine the reduction function according to the required target; Calculating a target feature using the reduction function based on the mapping tuple; The target feature is added to the mapping tuple, and processed using a comprehensive function to obtain the feature sequence.
6. The method for extracting traffic characteristics based on a programmable network according to claim 1, characterized in that: The steps for obtaining the feature vector are: The feature sequence is converted into the feature vector according to preset conversion parameters.
7. The method for extracting traffic characteristics based on a programmable network according to claim 6, characterized in that: The conversion parameter is the generation frequency of the feature vector.
8. A traffic feature extraction system based on a programmable network, characterized in that: include: Feature extraction module, used to abstract data packets and obtain data packet tuples; The step of obtaining the data packet tuple is that the SuperFE interface abstracts the data packet using a key-value pair to obtain the data packet tuple; the key-value pairs in the data packet tuple are divided into two categories, one category uses the header field of the data packet as a key and the value of the data packet as a value; the other category uses the metadata of the data packet as a key and a value generated according to the metadata of the data packet as a value; A grouping module, configured to group the data packet tuples according to a specified granularity to obtain group tuples; A screening module, configured to screen the data packet tuple or the group tuple based on a preset screening condition to obtain a screening tuple; The feature processing module is used to map the screening tuple to generate a mapping tuple, perform aggregation calculation on the mapping tuple to obtain a feature sequence, and convert the feature sequence into a feature vector.