A packet deduplication method of a TAP device and the TAP device

Abstract

The application discloses a packet deduplication method of a TAP device and the TAP device, and belongs to the technical field of network communication. The method comprises the following steps: an IPFIX unit built in the TAP device receives IPFIX deduplication template configuration information and IPFIX flow table behavior action information; the IPFIX unit obtains corresponding packet information based on the IPFIX deduplication template configuration information, and directly discards the packet based on the IPFIX flow table behavior action information when it is determined that the packet is not the first packet based on the packet information. The application can realize hardware deduplication based on the IPFIX unit built in the device, does not need to add new hardware, does not occupy CPU resources, can support high-bandwidth line-speed processing, greatly reduces the deployment cost of the deduplication function of the TAP device, and significantly improves the application scene and business carrying capacity of the device.

Description

Technical Field

[0001] This invention relates to the field of network communication technology, and in particular to a message deduplication method for a TAP device and the TAP device itself. Background Technology

[0002] Network traffic splitter (TAP) is a core passive data acquisition device in network monitoring systems. It can perform lossless copying of network traffic in transmission and perform filtering, aggregation, and distribution based on flow characteristics. It outputs specified traffic to systems such as network security auditing and business visualization analysis, and has become an essential deployment component for critical information infrastructure in finance, telecommunications, government and enterprise sectors.

[0003] In a multi-level TAP deployment architecture, the access layer TAP receives mirrored traffic from the collection points of each switch. After preliminary filtering, it is sent to the aggregation layer TAP. At this time, there will be a large number of duplicate packets in the mirrored traffic of multiple links. If it is directly distributed downstream, it will not only occupy the link bandwidth, but also put unnecessary performance pressure on the backend analysis system. Therefore, the aggregation layer TAP must have the ability to deduplicat traffic, which is the performance bottleneck of the entire collection link.

[0004] Currently, there are two main technical solutions for implementing TAP deduplication:

[0005] (1) CPU DPDK software deduplication scheme, which sends all traffic to the CPU through the chip, develops software deduplication logic based on the DPDK framework, uses packet information as feature field, and filters packets with duplicate features within a fixed time window (usually 100ms). This scheme is simple to implement and has low hardware modification cost, but the deduplication performance depends entirely on the CPU computing power. However, the general-purpose CPUs configured in TAP devices usually have limited performance and cannot support the high traffic processing needs of the aggregation layer TAP, resulting in a significant performance bottleneck in high bandwidth scenarios.

[0006] (2) FPGA hardware deduplication solution, which uses the programmable logic of FPGA to implement the deduplication function. Feature matching and table entry management are both completed at the hardware level, which can achieve line-speed processing. However, the FPGA device itself is expensive, which will significantly increase the overall procurement cost of TAP equipment, which is not conducive to large-scale deployment and promotion, and the actual implementation threshold is high.

[0007] In summary, neither of the existing deduplication solutions can simultaneously meet the deployment requirements of the aggregation layer TAP for high performance and low cost, which has become a technical pain point restricting the large-scale deployment of network traffic collection systems.

[0008] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to provide a message deduplication method and a TAP device, which can achieve hardware deduplication based on the IPFIX unit built into the TAP device, without the need for additional hardware, without occupying CPU resources, supporting high bandwidth line-speed processing, significantly reducing the deployment cost of deduplication function, and effectively improving the applicable scenarios and service carrying capacity of the device.

[0010] To achieve the above objectives, embodiments of the present invention provide a packet deduplication method for a TAP device, wherein the TAP device includes an IPFIX unit, and the method includes the following steps:

[0011] The IPFIX unit receives IPFIX deduplication template configuration information and IPFIX flow table behavior information;

[0012] The IPFIX unit obtains the corresponding packet information based on the IPFIX deduplication template configuration information, and directly discards the packet based on the IPFIX flow table behavior action information when it determines that the packet is not the first packet based on the packet information.

[0013] In one or more embodiments of the present invention, the IPFIX deduplication template configuration information includes

[0014] The 5-tuple field and the message identifier field; or

[0015] The five-tuple field, message identifier field, layer 4 message field, and payload field; or

[0016] The five-tuple field, message identifier field, payload field, IP fragmentation marker field, and IP fragmentation offset field, among which,

[0017] The five-tuple fields include the source IP address field, the destination IP address field, the network layer protocol type field, the layer 4 source port field, and the layer 4 destination port field.

[0018] In one or more embodiments of the present invention, the IPFIX deduplication template configuration information includes

[0019] Five-tuple field and stream label field; or

[0020] Five-tuple field, flow label field, layer 4 message field, and payload field.

[0021] In one or more embodiments of the present invention, it further includes:

[0022] When the IPFIX unit determines that a packet is the first packet based on the packet information, it directly forwards the packet based on the IPFIX flow table behavior action information.

[0023] In one or more embodiments of the present invention, a message is determined to be a non-first message by the following steps:

[0024] Determine the hash value based on the message information;

[0025] Search the hash table based on the hash value to determine if the corresponding entry exists.

[0026] If a corresponding entry exists, it is further determined whether the message information content of the two messages participating in the hash calculation is consistent.

[0027] If the message content is consistent, it is determined that the message is not the first message.

[0028] In one or more embodiments of the present invention, the method further includes

[0029] The IPFIX unit receives IPFIX flow aging time information and deletes the flow information in the table when the flow aging time is reached, based on the IPFIX flow aging time information.

[0030] In one or more embodiments of the present invention, at least one of the IPFIX deduplication template configuration information, IPFIX flow aging time information, and IPFIX flow table behavior action information is sent to the IPFIX unit via the CPU.

[0031] In one or more embodiments of the present invention, the IPFIX unit first receives IPFIX deduplication template configuration information, then receives IPFIX flow aging time information, and finally receives IPFIX flow table behavior action information.

[0032] Embodiments of the present invention provide a TAP device, including an IPFIX unit, wherein the IPFIX unit is configured to:

[0033] It receives IPFIX deduplication template configuration information and IPFIX flow table behavior information, obtains the corresponding packet information based on the IPFIX deduplication template configuration information, and directly discards the packet based on the IPFIX flow table behavior information when it is determined that the packet is not the first packet based on the packet information.

[0034] In one or more embodiments of the present invention, the IPFIX unit is further configured to: receive IPFIX flow aging time information, and delete the flow information in the table entry when the flow aging time is reached based on the IPFIX flow aging time information.

[0035] Compared with existing technologies, this invention can achieve hardware deduplication based on the IPFIX unit of existing chips without adding new hardware or occupying CPU resources. It can support high-bandwidth line-speed processing, significantly reduce the deployment cost of deduplication function of TAP devices, and significantly improve the applicable scenarios and service carrying capacity of the devices. Attached Figure Description

[0036] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the IPFIX unit structure according to an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of a TAP device according to an embodiment of the present invention. Detailed Implementation

[0039] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0040] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0041] like Figure 1 As shown, a message deduplication method for a TAP device according to a preferred embodiment of the present invention can achieve hardware deduplication based on the IPFIX unit built into the TAP device. It does not require additional hardware, does not occupy CPU resources, can support high bandwidth line-speed processing, significantly reduces the deployment cost of the TAP device's deduplication function, and significantly improves the device's applicable scenarios and service carrying capacity.

[0042] like Figure 1 As shown, a deduplication method for TAP devices includes the following steps:

[0043] S100, the IPFIX unit receives IPFIX deduplication template configuration information and IPFIX flow table behavior information;

[0044] Specifically, the TAP device includes an IPFIX (IP Flow Information Export) unit. The IPFIX unit can be used to extract and learn the IP flow characteristics of network packets, enabling network traffic status monitoring and network fault location. This invention reuses the built-in IPFIX unit of the TAP device to achieve packet deduplication processing, eliminating the need for additional hardware devices (such as FPGA devices) and avoiding the consumption of device CPU processing resources. While supporting high-bandwidth line-rate traffic deduplication, it significantly reduces the deployment cost of the TAP device's deduplication function.

[0045] During packet deduplication, the IPFIX unit receives IPFIX deduplication template configuration information. This configuration information instructs the IPFIX unit to perform deduplication on packets that meet matching conditions. By configuring different IPFIX deduplication template configurations, the IPFIX unit can adapt to the deduplication requirements of different types of packets, expanding the applicable scenarios of the deduplication scheme. For example, when deduplication is required for IPv4 packets, the IPFIX deduplication template configuration information includes a 5-tuple field and a packet ID field, or a 5-tuple field, a packet ID field, a Layer 4 packet field, and a payload field. Alternatively, for fragmented packets, the IPFIX deduplication template configuration information includes a 5-tuple field, a packet ID field, a Layer 4 packet field, a payload field, an IP fragmentation marker field (IP MF), and an IP fragmentation offset field. The configuration can be tailored to specific needs. The 5-tuple fields include the source IP address, destination IP address, network layer protocol type, layer 4 source port, and layer 4 destination port. The layer 4 packet fields include the layer 4 checksum (l4_checksum), TCP sequence number (tcp_seq_number), TCP acknowledgment number (tcp_ack_number), and TCP flags (tcp_flags). The payload fields include the packet payload and packet length (packet_len). For example, when deduplication of IPv6 packets is required, the IPFIX deduplication template configuration information includes the 5-tuple fields and flow label fields; or the 5-tuple fields, flow label fields, layer 4 packet fields, and payload fields, which can be configured according to actual needs. The 5-tuple fields include the source IP address, destination IP address, network layer protocol type, layer 4 source port, and layer 4 destination port. The Layer 4 packet fields include the Layer 4 checksum (l4_checksum), TCP sequence number (tcp_seq_number), TCP acknowledgment number (tcp_ack_number), and TCP flags (tcp_flags). The payload fields include the packet payload and packet length (packet_len).

[0046] Meanwhile, during the packet deduplication process, in addition to receiving IPFIX deduplication template configuration information, the IPFIX unit also receives IPFIX flow table behavior information. This IPFIX flow table behavior information can be used to configure what operations IPFIX performs on packets, such as discarding duplicate packets. In this embodiment, the IPFIX flow table behavior information includes performing a continue forwarding action for the first packet and a packet discarding action for subsequent packets.

[0047] In this embodiment, the IPFIX deduplication template configuration information and the IPFIX flow table behavior information can be received by the IPFIX unit simultaneously. Of course, in other embodiments, the IPFIX unit can also receive the IPFIX deduplication template configuration information first and then the IPFIX flow table behavior information, which can be set according to actual needs.

[0048] S200, the IPFIX unit obtains the corresponding packet information based on the IPFIX deduplication template configuration information, and directly discards the packet based on the IPFIX flow table behavior action information when it is determined that the packet is not the first packet based on the packet information.

[0049] Specifically, after the TAP device receives a packet, the IPFIX unit can extract the corresponding packet information based on the IPFIX deduplication template configuration information. For example, for IPv4 packets, the IPFIX deduplication template configuration information includes a 5-tuple field, a packet identifier field, a Layer 4 packet field, a payload field, an IP fragmentation marker field (IP MF), and an IP fragmentation offset field (IPFragment Offset). When the TAP device receives an IPv4 packet, it parses the packet to obtain the corresponding 5-tuple field, packet identifier field, Layer 4 packet field, payload field, and IP fragmentation marker field carried by the packet.

[0050] Furthermore, after obtaining the message information, it is further determined whether the message is the first message based on that information. In specific implementation, the following steps can be used to determine whether a message is the first message:

[0051] First, the hash value is determined based on the message information;

[0052] Secondly, the hash value is used to search the hash table to determine if the corresponding entry exists.

[0053] Secondly, if no corresponding entry exists, it is determined to be the first message; if a corresponding entry exists, it is further determined whether the message information content of the two messages participating in the hash calculation is consistent.

[0054] Finally, if the message content is consistent with the response, it is determined to be a non-first message; if the message content is inconsistent with the response, it is determined to be the first message.

[0055] like Figure 2 As shown, the IPFIX unit includes an inbound processing module, an IPFIX engine module, an IPFIX entry module, and an outbound processing module. Upon receiving a packet, the inbound processing module processes the packet and then sends it to the IPFIX engine module. The IPFIX engine module parses the packet using its internal packet parsing module to obtain packet information, such as the corresponding five-tuple field, packet identifier field, Layer 4 packet field, payload field, and IP fragmentation tag field. After obtaining the packet information, it sends it to the internal hash calculation unit for calculating a hash value based on the packet information.

[0056] After calculating the hash value, the system further searches the hash table in the IPFIX entry module to determine if an entry with the same hash value exists. This hash table is used to store flow information. If no entry with the same hash value exists in the hash table, the packet is identified as the first packet. Based on the IPFIX flow table action information, the packet is forwarded normally, and the hash value is used as the key to store the packet header information in the hash table.

[0057] When it is determined that entries with the same hash value exist in the hash table, it indicates a hash collision. To avoid misjudging it as a non-first packet, it is necessary to further determine whether the packet information content involved in the hash calculation of the two packets is consistent. That is, whether the packet information obtained based on the IPFIX deduplication template configuration information is consistent. If the packet information is consistent, it is determined to be a non-first packet, and the packet is a duplicate packet. Therefore, based on the content of the IPFIX flow table behavior action information, the packet is discarded. If the packet information is inconsistent, it is determined to be the first packet, and the packet is forwarded normally. At the same time, the conflicting packet is sent to the TCAM table entry.

[0058] Furthermore, in this embodiment, the IPFIX unit also receives IPFIX flow aging time information. This IPFIX flow aging time information can be used to configure the flow information aging time. In specific implementations, the IPFIX unit can automatically delete the flow information in the hash table when the flow aging time is reached based on the IPFIX flow aging time information, thereby improving the resource utilization of the hash table. By continuously learning flow information and aging table entries, it is possible to determine whether the flow information is a duplicate flow within a specific time period, thus achieving the deduplication function.

[0059] The IPFIX flow aging time information here can be received by the IPFIX unit simultaneously with the IPFIX deduplication template configuration information and IPFIX flow table behavior information mentioned above. Alternatively, the IPFIX deduplication template configuration information can be received first, followed by the IPFIX flow aging time information, and finally the IPFIX flow table behavior information. This can be configured according to actual needs.

[0060] like Figure 3 The diagram shows a TAP device disclosed in this invention, capable of implementing the aforementioned TAP device deduplication method. The TAP device includes an IPFIX unit, configured to receive IPFIX deduplication template configuration information and IPFIX flow table behavior information, obtain corresponding packet information based on the IPFIX deduplication template configuration information, and directly discard the packet based on the IPFIX flow table behavior information when it is determined that the packet is not the first packet based on the packet information.

[0061] How the IPFIX unit implements the corresponding functions is detailed above and will not be repeated here.

[0062] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for deduplicating messages in a TAP device, the TAP device comprising an IPFIX unit, characterized in that, The method includes the following steps: The IPFIX unit receives IPFIX deduplication template configuration information and IPFIX flow table behavior information; The IPFIX unit obtains the corresponding packet information based on the IPFIX deduplication template configuration information, and directly discards the packet based on the IPFIX flow table behavior action information when it determines that the packet is not the first packet based on the packet information.

2. The method as described in claim 1, characterized in that, The IPFIX deduplication template configuration information includes The 5-tuple field and the message identifier field; or The five-tuple field, message identifier field, layer 4 message field, and payload field; or The five-tuple field, message identifier field, payload field, IP fragmentation marker field, and IP fragmentation offset field, among which, The five-tuple fields include the source IP address field, the destination IP address field, the network layer protocol type field, the layer 4 source port field, and the layer 4 destination port field.

3. The method as described in claim 1, characterized in that, The IPFIX deduplication template configuration information includes Five-tuple field and stream label field; or Five-tuple field, flow label field, layer 4 message field, and payload field.

4. The method as described in claim 1, characterized in that, The method also includes: When the IPFIX unit determines that a packet is the first packet based on the packet information, it directly forwards the packet based on the IPFIX flow table behavior action information.

5. The method as described in claim 1, characterized in that, The following steps can be used to determine if a message is not the first message: Determine the hash value based on the message information; Search the hash table based on the hash value to determine if the corresponding entry exists. If a corresponding entry exists, it is further determined whether the message information content of the two messages participating in the hash calculation is consistent. If the message content is consistent, it is determined that the message is not the first message.

6. The method as described in claim 1, characterized in that, The method also includes The IPFIX unit receives IPFIX flow aging time information and deletes the flow information in the table when the flow aging time is reached, based on the IPFIX flow aging time information.

7. The method as described in claim 6, characterized in that, At least one of the IPFIX deduplication template configuration information, IPFIX flow aging time information, and IPFIX flow table behavior action information is sent to the IPFIX unit via the CPU.

8. The method as described in claim 6, characterized in that, The IPFIX unit first receives IPFIX deduplication template configuration information, then receives IPFIX flow aging time information, and finally receives IPFIX flow table behavior information.

9. A TAP device, characterized in that, The system includes an IPFIX unit, which is configured to: receive IPFIX deduplication template configuration information and IPFIX flow table behavior action information; obtain corresponding packet information based on the IPFIX deduplication template configuration information; and directly discard the packet based on the IPFIX flow table behavior action information when it is determined that the packet is not the first packet based on the packet information.

10. The device as claimed in claim 9, characterized in that, The IPFIX unit is further configured to receive IPFIX flow aging time information and delete the flow information in the table entry when the flow aging time is reached, based on the IPFIX flow aging time information.

Images