A scalable high performance network border traffic distribution system and method

Abstract

The application relates to the technical field of network supervision, and discloses an extensible high-performance network boundary traffic distribution system and method, which comprises the following steps: a configuration proxy module analyzes network configuration information, and respectively sends the network configuration information of a collection port and a forwarding port to a traffic collection module and a traffic filtering and forwarding module; a policy proxy module analyzes network policy information, and sends filtering policy and forwarding policy to a message identification module and the traffic filtering and forwarding module; the traffic collection module collects data messages after setting the collection port based on the network configuration information of the collection port, and forwards the traffic to the message identification module or the traffic filtering and forwarding module based on the start-stop condition of the message identification module; the message identification module labels the traffic after filtering the traffic based on the filtering policy; and the traffic filtering and forwarding module forwards the traffic according to the forwarding policy and the network configuration of the forwarding port after filtering the traffic based on the filtering policy. The system can accurately control the fine granularity of traffic forwarding.

Description

Technical Field

[0001] The present invention relates to the technical field of network supervision, and in particular to an extensible high-performance network edge traffic distribution system and method. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In today's information society, network security has become increasingly important. As the complexity and frequency of cyber attacks continue to increase, enterprises, institutions and individuals are facing serious threats such as data leakage, malware infection and cyber attacks. Therefore, it is necessary to protect against dangerous behaviors such as cyber attacks and to regulate the compliance of network activities.

[0004] In the process of network security prevention, control and management, due to the diversity of network traffic, understanding and processing network traffic becomes more critical. Different types of traffic may need to go through different processing devices or different processing methods.

[0005] Therefore, pre-processing of network traffic becomes critical. In the pre-processing stage, a dedicated traffic classifier is needed to identify, filter, classify and distribute traffic. The scalable, high-performance network edge traffic distribution system comes into being, aiming to solve complex traffic detection problems and achieve effective monitoring and security detection of large-scale network traffic.

[0006] Existing network edge traffic distribution systems are generally based on DPDK and OVS (Open vSwitch) or DPDK+VPP.

[0007] The main functions of these network edge traffic distribution systems include:

[0008] Flexible and efficient traffic collection: The system supports kernel-mode and user-mode collection engines, utilizing the kernel-mode AF_PACKET collection engine and the DPDK user-mode collection engine for traffic collection.

[0009] Policy rule filtering and forwarding: Analyze and decide whether to allow or deny specific data packets based on the set policy. This includes filtering based on IP addresses, port numbers, protocol types, and session flow status. Some systems also perform deep packet filtering based on DPI to filter out dangerous and sensitive information, and forward packets based on the filtered data. Forwarding mainly determines the forwarding path of the data packet based on the destination address of the data packet to ensure that the data packet can reach the destination address from the source address.

[0010] Efficient traffic forwarding: Enables fast packet forwarding and processing, supports batch processing of packets, and efficiently handles large amounts of concurrent traffic, thereby improving overall system throughput and providing efficient distribution capabilities for back-end business processing.

[0011] These network edge traffic distribution systems generally have the following problems:

[0012] (1) Performance bottleneck: Low performance is the main disadvantage of software forwarding. If it is not based on hardware conditions, forwarding may become a bottleneck in a high-traffic environment, resulting in increased latency and decreased throughput, especially when processing a large amount of data packets.

[0013] (2) Resource consumption: The granularity of filtering is relatively coarse. In fact, traffic forwarding requires the consumption of CPU and memory resources, especially when a large number of data packets need to be processed, which may lead to excessive occupation of system resources. In high-traffic scenarios, higher CPU and memory resources may be required. In particular, when performing deep packet processing, resource consumption will increase significantly.

[0014] (3) Poor scalability: These forwarding systems are generally completely separated from the business system, do not uniformly manage business policies, and have relatively scattered business targeting. At the same time, these systems are generally deployed on a single device for business modules. For large-scale networks, especially those involving high throughput and large-scale concurrent connections, the traffic forwarding of such systems often lacks good scalability. Summary of the Invention

[0015] In order to solve the above problems, the present invention provides a scalable high-performance network edge traffic distribution system and method to solve the problems of pre-filtering and distribution of traffic. According to different back-end integration requirements, the DPI-related functional modules can be started and stopped to accurately control the fine-grained traffic forwarding, providing support for subsequent business to handle large traffic.

[0016] In order to achieve the above object, the present invention adopts the following technical solutions:

[0017] A first aspect of the present invention provides a scalable, high-performance network edge traffic distribution system, comprising a control plane and a data plane disposed in different central processing units or hosts; the control plane comprises a configuration agent module and a policy agent module; the data plane comprises a traffic collection module, a message identification module, and a traffic filtering and forwarding module, and the message identification module is enabled or disabled according to different backend integration requirements;

[0018] The configuration proxy module is used to parse the network configuration information and send the network configuration information of the collection port and forwarding port to the traffic collection module and the traffic filtering and forwarding module respectively;

[0019] The policy agent module is used to parse the network policy information, send the filtering policy to the message identification module, and send the filtering policy and forwarding policy to the traffic filtering and forwarding module;

[0020] The traffic collection module is used to collect data messages after the collection port is set based on the network configuration information of the collection port, and forward the traffic to the message identification module or the traffic filtering and forwarding module based on the start and stop status of the message identification module;

[0021] The message identification module is used to filter the traffic based on the filtering policy and then add a label;

[0022] The traffic filtering and forwarding module is used to filter the traffic based on the filtering policy and then forward the traffic according to the forwarding policy and the network configuration of the forwarding port.

[0023] Furthermore, it also includes a data plane control agent, and the data plane and the data plane control agent are set in the same central processor or host;

[0024] The data plane control agent is used to send network configuration information and network policy information.

[0025] Furthermore, the control plane also includes message middleware;

[0026] The message middleware is used to push network configuration information and network policy information to the configuration agent module and the policy agent module respectively.

[0027] Furthermore, the traffic collection module collects traffic from the network port in a polling or interruption manner.

[0028] Furthermore, the traffic collection module starts a user state or kernel state collection engine according to the configured traffic collection mode, and switches to a polling mode or an interrupt mode to collect traffic according to the size of the traffic.

[0029] Furthermore, the network configuration information includes the configuration of network IP, network card binding, collection port and forwarding port.

[0030] Furthermore, the message identification module adopts DPI protocol identification and separates the feature library from the engine.

[0031] Furthermore, the traffic filtering and forwarding module filters the traffic including packet filtering, flow filtering, session filtering, application filtering and network traffic audit policy filtering;

[0032] Furthermore, the traffic filtering and forwarding module supports Layer 2 forwarding, IP routing forwarding and multi-group combination strategies.

[0033] A second aspect of the present invention provides a traffic distribution method for a scalable, high-performance network edge traffic distribution system based on the first aspect, comprising the following steps:

[0034] The configuration proxy module parses the network configuration information and sends the network configuration information of the collection port and forwarding port to the traffic collection module and traffic filtering and forwarding module respectively;

[0035] The policy agent module parses the network policy information and sends the filtering policy to the message identification module, and sends the filtering policy and forwarding policy to the traffic filtering and forwarding module;

[0036] After the traffic collection module sets up the collection port based on the network configuration information of the collection port, it collects data packets and forwards the traffic to the message identification module or the traffic filtering and forwarding module based on the start and stop status of the message identification module;

[0037] The message identification module filters the traffic based on the filtering policy and adds a label;

[0038] The traffic filtering and forwarding module filters the traffic based on the filtering policy, and then forwards the traffic according to the forwarding policy and the network configuration of the forwarding port.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention provides a scalable, high-performance network edge traffic distribution system that solves the problems of pre-filtering and distribution of traffic. According to different back-end integration requirements, it can start and stop DPI-related functional modules to accurately control the fine-grained traffic forwarding, providing support for subsequent business to handle large traffic flows.

[0041] The present invention provides a scalable, high-performance network edge traffic distribution system that performs traffic filtering and forwarding control through configuration and policies. In specific scenarios, full traffic forwarding is required. The control plane can control the component functions of the data plane and start different module functions as needed.

[0042] The present invention provides a scalable, high-performance network edge traffic distribution system. It applies DPDK technology in the traffic collection part, optimizes the packet receiving architecture, and maximizes the system's traffic collection performance. It also applies VPP technology in the data plane traffic forwarding part, greatly improving the packet filtering and forwarding capabilities through functional expansion based on VPP. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention but do not constitute a limitation of the present invention.

[0044] Figure 1 This is a schematic diagram of the separation of the control plane and the data plane in accordance with the first embodiment of the present invention;

[0045] Figure 2 This is a flow chart of the flow collection process according to the first embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the pre-flow filtering policy control according to the first embodiment of the present invention;

[0047] Figure 4 This is a DPI data processing flow chart of Example 1 of the present invention;

[0048] Figure 5 Schematic diagram of VPP forwarding plane pipeline processing according to the first embodiment of the present invention;

[0049] Figure 6 This is a logical diagram of a traffic distribution framework according to the first embodiment of the present invention;

[0050] Figure 7 This is an overall architecture diagram of the network boundary traffic detection framework of the first embodiment of the present invention;

[0051] Figure 8 This is a processing sequence diagram of the system according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] In the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0056] Example 1

[0057] The purpose of this embodiment 1 is to provide a scalable, high-performance network edge traffic distribution system.

[0058] This embodiment provides a scalable, high-performance network edge traffic distribution system that provides efficient and accurate fine-grained control and forwarding of network traffic, filtering out unnecessary network traffic. Key features include control and data plane separation, a flexible traffic collection engine, policy rule filtering and management, high-performance traffic forwarding, and system monitoring and management.

[0059] This embodiment provides a scalable, high-performance network edge traffic distribution system, which has the following main functions:

[0060] Flexible and efficient traffic collection: The system supports kernel-mode and user-mode collection engines, using the traditional kernel-mode AF_PACKET collection engine and the DPDK user-mode collection engine for traffic collection. The traffic collection mode can be switched through the API interface.

[0061] Unified network configuration management: Centrally manage the network configuration of different components and sub-services to achieve docking coordination and resource consistency within the host and cross-host network environments.

[0062] Policy rule filtering and forwarding: Based on the multi-dimensional traffic access control list mechanism, it forms a multi-dimensional traffic filtering capability, supporting event-level control, blacklist and whitelist control, application-level control, session-level control and user-level control; the rule filtering engine implements layer-by-layer offloading processing of collected traffic, including packet filtering, flow filtering, session filtering, application filtering, network traffic audit policy filtering, etc., and through rich filtering rules, hierarchical traffic filtering and offloading acceleration processing.

[0063] Front-end DPI deep packet analysis: Through DPI deep packet analysis, intelligent traffic control is performed, DPI message analysis is coordinated with secure forwarding strategies to optimize back-end traffic quality, filter out unnecessary traffic messages, and provide session-level and application-level traffic control capabilities.

[0064] VPP high-performance data processing plane: It achieves fast packet forwarding and processing, supports batch processing of packets, and efficiently handles large amounts of concurrent traffic, thereby improving overall system throughput and providing efficient distribution capabilities for back-end business processing.

[0065] This embodiment provides a scalable high-performance network edge traffic distribution system, such as Figure 1 As shown, the system includes a control plane, a data plane, and a data plane control agent (also known as a control agent module). The control plane and data plane are located in different central processing units (CPUs) or hosts, while the data plane and data plane control agent are located in the same CPU or host. Separating the control plane from the data plane reduces functional coupling, enables independent maintenance, reduces complexity, lowers maintenance requirements, and minimizes the scope of failure domains.

[0066] Among them, the control plane is responsible for system initialization, configuration changes, policy updates, and data queries.

[0067] Among them, the data plane is responsible for continuous data capture and analysis.

[0068] The control plane and data plane can exist and upgrade independently; the control plane and data plane can run independently and flexibly be deployed.

[0069] Among them, such as Figure 7 As shown in Figure 1, the control plane ZNMS manages and issues configurations, policies, and commands. The control plane includes a message middleware, a configuration agent module, and a policy agent module.

[0070] ZNMS uses Kafka as a messaging middleware to distribute relevant control and business data to the traffic collection and forwarding modules. Strategies include filtering and forwarding based on blacklists and whitelists, as well as IP, port, and protocol filtering. Control commands include host startup and shutdown, module startup and shutdown, and sub-service status monitoring.

[0071] The policy proxy module, ZNDC-Sub-Server (also known as the API policy receiving module), receives policies from the message middleware, parses them, and forwards them. The policy proxy module primarily parses specific policies issued by the management module, reassembles them based on the module number assigned to them, and forwards them to the corresponding module or sub-service. Policies are used to control functions, configure interfaces, and implement specific policy actions.

[0072] Among them, the configuration agent module (also known as the configuration management module) can uniformly configure the forwarding framework according to specific business needs, and the configuration management module can uniformly control the operation of the entire business system, and can monitor the business operation status, data processing format, etc.

[0073] The data plane includes a traffic collection and forwarding module. The traffic collection and forwarding module ZNDC integrates the DPDK packet receiving module (also known as the traffic collection module), the DPI deep packet inspection module (also known as the message identification module), and the VPP high-speed traffic forwarding module (also known as the traffic filtering and forwarding module). It supports cross-host network port forwarding and zero-copy shared queue distribution on the same host. At the same time, it also supports virtual network port traffic forwarding on the same host, such as Figure 6 shown.

[0074] Among them, the traffic collection module adopts a flexible traffic collection engine: used for collecting, sharing, ACL forwarding, etc. of network traffic; it achieves high-performance packet capture through zero-copy technology, and can receive real-time network traffic, replay traffic, or directly read PCAP files.

[0075] Among them, such as Figure 2As shown, the traffic collection module is configured as follows: configure the traffic collection mode; start the user state or kernel state collection engine according to the configured traffic collection mode; flexibly switch events or polling mechanisms to collect traffic based on traffic size; collect all traffic and obtain traffic data packets from the network card queue; send traffic to the processing queue for subsequent BPI analysis, policy matching and forwarding.

[0076] Among them, the traffic collection module can control the data access method (traffic collection mode) through configuration, including DPDK traffic collection, PF_RING traffic collection, AF_PACKET traffic collection and LIBPCAP library-based traffic collection, and intelligently set relevant configurations according to specific business performance requirements.

[0077] The packet identification module uses the DPI protocol for identification. The DPI packet identification module separates the signature library from the engine processing, making the signature library independent of the system engine. The signature library compilation module analyzes application signatures and compiles them into signature library files. The main engine processing module loads and parses the signature library files, while also matching and processing data packets.

[0078] Among them, the data packet parsing of the packet identification module supports L2-L7 layer protocol identification and matching, and can filter traffic based on IP, port, application layer protocol, application name, session flow, etc.

[0079] Currently, the DPI layer supports processing of TCP / UDP traffic and the data processing flow of the message identification module, such as Figure 4As shown, it includes: (1) TCP / UDP judgment: check whether the traffic is TCP or UDP protocol; if not (N), go to step (12); if yes (Y), go to step (2). (2) Hash table query: perform hash table query from the flow through the five-tuple (source IP, source port, destination IP, destination port, protocol). If the query is found, go to step (3); if not, go to step (4). (3) Query the mark in the flow to determine whether the flow is terminated; if not, go to step (10); if it is terminated, determine whether there is an identification result. If there is an identification result, determine whether the maximum number of processing is reached. If the maximum number of processing is reached, go to step (10); if there is no identification result or the maximum number of processing is not reached, go to step (5). (4) Create a new flow and enter the matching process. If there is a blacklist before matching, perform blacklist matching first and determine whether the blacklist filtering is successful. If successful (Y), the process ends; if unsuccessful (N), go to step (5). (5) Pre-filter matching: Pre-filter matching is performed by setting the filter hash table query; if the match is successful (Y), go to step (10); if the match is unsuccessful (N), go to step (6). (6) Pre-grouping: Pre-group the traffic, including HTTP groups, TLS groups, TCP groups and UDP groups, to speed up the matching speed. (7) HS regular matching: Perform HS regular matching and return the regular matching rule ID and the matching content location. (8) High-speed pattern matching module matching: Through the rule ID result matched by the high-speed pattern matching module, reverse search the corresponding protocol classifier, and then match according to the protocol classifier. (9) Protocol information extraction: Extract protocol information from the identified traffic. (10) Post-processing: Post-process the traffic that has been identified or not identified but has reached the maximum depth. (11) IANA protocol matching, DNS setting filter, IP blacklist setting: After post-processing, perform IANA protocol matching, DNS setting filter and IP blacklist setting. (12) Forwarding: Finally, the processed traffic is forwarded.

[0080] The traffic filtering and forwarding module is configured with a pre-placed policy rule filtering submodule and a traffic forwarding control submodule.

[0081] like Figure 3 As shown in the figure, the policy rule filtering submodule performs layer-by-layer offloading processing on the collected traffic, including packet filtering, flow filtering, session filtering, application filtering, and network traffic audit policy filtering. Through rich filtering rules, hierarchical traffic filtering, and offload acceleration processing design, it can effectively reduce the complexity of protocol depth analysis and calculation, and can accurately and effectively filter unnecessary traffic and screen out the traffic messages required by the business system.

[0082] The traffic forwarding control submodule (also known as the traffic control and forwarding module) supports Layer 2 forwarding, including Layer 2 forwarding capabilities for VLAN and QinQ packets; supports IP routing forwarding, including support for IPv4 and IPv6; supports multi-group combination strategies, including but not limited to IP blacklists and whitelists, ports, protocols, etc., and can perform packet filtering, distribution, redirection and other operations.

[0083] Among them, the traffic forwarding control submodule supports cross-host and intra-host inter-process traffic distribution. The intra-host is mainly for single device resource scenarios, integrating back-end services into the same business system; cross-host is only for scenarios with high-traffic business scenarios where single device resources are limited and processing performance cannot meet actual needs.

[0084] In order to support flexible traffic filtering functions, support multi-dimensional traffic filtering strategies, and improve the processing performance of traffic collection and forwarding modules, the next step of distribution processing will be carried out only when the strategy matches and the action is Permit.

[0085] VPP forwarding plane flow processing is as follows Figure 5 As shown, VPP modifies and expands the original pipeline:

[0086] (1) Use the ACL plugin as the basic ACL policy control, but need to extend the ACL action to associate the action profile description configuration; and add the session node / DPI node / distribution node / IP fragment pre-reassembly node. The newly added nodes are dynamically controlled and inserted using the API;

[0087] (2) Distribution policy control based on ACL+DPI allows specific 5-tuple (Layer 3 protocol) messages + specific applications / Layer 4 protocols / URLs to be distributed to specific targets for further security analysis and processing;

[0088] (3) The distribution policy can be configured to support 8-tuples, 4-layer port association, and ACL rules for IP fragment aggregation with the same SRCIP+DSTIP+IPID (fragment ID). Packet aggregation search, aggregation processing, and forwarding can be achieved through IP fragment pre-reassembly.

[0089] (4) In vlib_buffer_t, construct the opaque3 context information field. The opaque3 field is used to transmit context information between nodes in the arc chain. The main information includes: the five-tuple information corresponding to the packet, the policy rule number and the associated action profileid, the policy application ID, the flag bits that need to be transmitted between nodes, NAT information, etc.

[0090] (5) The distribution node is responsible for target redirection distribution and performs secure application redirection distribution according to the configured load sharing strategy, target list and NAT strategy and the distribution action profile description configuration;

[0091] (6) The session node accelerates the subsequent data of the same flow by recording the five-tuple of each flow, as well as the policy rule number corresponding to the flow, the associated action profile ID, the policy application ID, the processing action of the subsequent node, and other information. When the NAT rule is issued, the session node can also record the NAT reverse session;

[0092] (7) The policy node / DPI node / distribution node provides the corresponding binary API interface to the upper layer for configuration;

[0093] (8) IP6 water flow is similar to IP4.

[0094] VPP forwarding plane flow processing Figure 5 As shown, where:

[0095] Packet vector: This is the input source of the data packet, which contains multiple input interfaces, such as P1, P2, P3, etc. These interfaces may represent different network interface cards (NICs).

[0096] Starting from the Packet vector, there are multiple input paths connected to different processing nodes. Processing nodes and their functions:

[0097] af - packet - input: Receives packets from the Packet vector, which may be input based on the AF_PACKET mechanism;

[0098] mpls-input: processes MPLS (Multi-Protocol Label Switching) packets;

[0099] ethernet-input: processes Ethernet packets;

[0100] dpdk - input: Receives packets using DPDK (Data Plane Development Kit), typically used for high-performance packet processing;

[0101] l2 - input: processes Layer 2 (data link layer) packets;

[0102] l2-dispatch: dispatches and processes layer 2 data packets;

[0103] ip6 - input: processes IPv6 packets;

[0104] llc - input: processes LLC (Logical Link Control) packets;

[0105] pppoe - input: handles PPPoE (Point-to-Point Protocol over Ethernet) packets;

[0106] arp - input: Processes ARP (Address Resolution Protocol) packets.

[0107] Core processing path:

[0108] ip4 - input: processes IPv4 packets;

[0109] ip4 - sv - reassembly - policy: performs segmentation and reassembly policy processing on IPv4 packets;

[0110] ip4 - policy - input: applies IPv4 policy input;

[0111] ip4 - zdpi - handoff: passes the packet to the ZDPI (Zero Day Packet Detection and Prevention) module;

[0112] ip4 - zdpi: performs ZDPI processing;

[0113] ip4 - zdpi - gtpu: handles GTP-U (GPRS Tunneling Protocol - User Plane) packets;

[0114] ip4 - zdpi - capwap: handles CAPWAP (Control and Configuration of Wireless Access Points Protocol) packets;

[0115] ip4-lookup: performs lookup operations on IPv4 packets;

[0116] ip4 - cat - x: performs some sort of classification operation (perhaps based on categories or tags);

[0117] ip4-dispatch: distributes IPv4 packets;

[0118] ip4 - buffer - duplication: duplicate IPv4 packet buffer;

[0119] ip4 - fa: This may be some kind of forwarding action.

[0120] ip4 - cat - snat - static: performs static source NAT (Network Address Translation) operations;

[0121] ip4-rewrite: rewrite IPv4 packets;

[0122] ip4 - policy - output: applies IPv4 policy output;

[0123] zach - plugin - in - ip4 - fa: Apply a plugin (possibly a Zach plugin) to perform IPv4 forwarding operations;

[0124] zach - plugin - out - ip4 - fa: Output after applying a plugin to perform IPv4 forwarding operations;

[0125] ip4 - firewall: performs IPv4 firewall operations;

[0126] nat44_session_handoff: handles NAT44 (Network Address Translation version 44) session handoff;

[0127] interface - output: The processed data packets are finally output to the network interface.

[0128] like Figure 8 As shown in the figure, the data flow between the control agent module, message middleware, configuration agent module, policy agent module, DPDK packet receiving module (also known as traffic collection module), DPI deep packet inspection module (also known as message identification module), and VPP high-speed traffic forwarding module (also known as traffic filtering and forwarding module) includes the following steps:

[0129] The control plane's control agent (data plane control agent) distributes network configuration information to a designated configuration topic. This information includes basic network IP addresses, network interface card bindings, and the configuration of collection and forwarding ports (port groups and ports). This distributed network configuration information is then pushed to the configuration agent via the messaging middleware. The configuration agent parses the information and calls APIs to set the network configuration. The configuration agent then distributes the collection port network configuration information to the traffic collection module and the forwarding port network configuration information (port group and port configuration information) to the VPP traffic filtering and forwarding module. The control plane's control agent distributes network policy information to a designated configuration topic. This distributed network policy information is then pushed to the policy agent via the messaging middleware. After parsing the policy information, the policy agent sends the application-layer filtering policy to the DPI packet identification module. The policy agent then sends the filtering and forwarding policies to the VPP high-speed traffic forwarding module. The DPI packet identification module identifies the application-layer protocol and, if relevant, tags the traffic to facilitate subsequent VPP policy filtering and forwarding. After the traffic collection module identifies the configured collection port and sets the collection mode and port based on the port's network configuration information, it starts collecting data packets from the network port in either a polling or interruption mode. Based on the packet identification module's start / stop status, it forwards the traffic to the DPI packet identification module or the VPP traffic filtering and forwarding module. The interruption or polling mode can be switched based on traffic volume, with interruption used for low-volume traffic and polling used for high-volume traffic. Traffic is sent to the DPI module via a shared queue. The DPI deep packet processing module extracts traffic from the shared queue for in-depth analysis after the traffic collection module forwards it to the shared queue. It filters the traffic based on the filtering policy and tags matching packets (traffic to be forwarded). If DPI is disabled, traffic is sent to the VPP module via a shared queue. The VPP filters the traffic sent from the collection module and then forwards the relevant traffic to the designated queue or network port based on the forwarding policy and the network configuration of the forwarding port. Queue mode is based on intra-host forwarding, while network port mode is based on inter-host traffic forwarding. This embodiment provides a scalable, high-performance network edge traffic distribution system. By optimizing hardware resources, it uses DPDK technology to optimize the packet receiving architecture and provide an interface for hardware resource configuration, ensuring that the system obtains appropriate hardware resources at the packet receiving level. Furthermore, through resource partitioning and contention elimination, the system's performance scalability is enhanced as hardware resources increase, supporting 20Gbps network traffic processing.

[0130] This embodiment provides a scalable, high-performance network edge traffic distribution system that adopts flexible policy control and supports pre- and post-policy control. In the traffic collection and distribution processing module, policies can be used to flexibly control traffic filtering and forwarding channels. In the traffic parsing and analysis module, policies can be used to control the fine-grainedness of traffic parsing and analysis.

[0131] This embodiment provides a scalable, high-performance network edge traffic distribution system that uses a load balancing mechanism to distribute traffic detection tasks, ensuring uniform load among distribution nodes.

[0132] This embodiment provides a scalable, high-performance network edge traffic distribution system that supports horizontal and vertical expansion and can dynamically increase or decrease nodes as needed to cope with traffic fluctuations.

[0133] This embodiment provides a scalable, high-performance network edge traffic distribution system that supports plug-in detection algorithms and analysis tools, facilitating the integration of new detection technologies and algorithms.

[0134] This embodiment provides a scalable, high-performance network edge traffic distribution system that primarily addresses traffic pre-filtering and distribution issues. It accurately controls the fine-grained nature of traffic forwarding based on different business scenarios, providing support for subsequent business systems to process large volumes of traffic.

[0135] This embodiment provides a scalable, high-performance network edge traffic distribution system. Since traffic filtering and forwarding control are performed through configuration and policies, full traffic forwarding is required in specific scenarios. It is necessary to control the component functions of the data plane on the control plane and start different module functions as needed.

[0136] This embodiment provides a scalable, high-performance network edge traffic distribution system. It applies DPDK technology in the traffic collection component to optimize the packet receiving architecture and maximize the system's traffic collection performance. It also applies VPP technology in the data plane traffic forwarding component. By expanding VPP-based functionality, it greatly enhances packet filtering and forwarding capabilities.

[0137] This embodiment provides a scalable, high-performance network edge traffic distribution system that can improve overall processing performance by starting and stopping DPI-related functional modules and providing only five-tuple traffic filtering functions according to different back-end integration requirements; it adopts a modular design and can flexibly adjust the deployment method.

[0138] This embodiment provides a scalable, high-performance network edge traffic distribution system. To reduce the difficulty of deployment and maintenance management, the overall filtering and distribution framework provides a unified operation and maintenance management API and visualization interface, and function configuration and module start and stop are performed through the API or visualization page.

[0139] Example 2

[0140] The purpose of this second embodiment is to provide a traffic distribution method based on the scalable high-performance network edge traffic distribution system described in the first embodiment, including the following steps:

[0141] Step 1: Send network configuration information to the specified configuration topic through the control agent module of the control plane, including basic network IP, network card binding, network port group and forwarding port configuration.

[0142] Step 2: The distributed network configuration information is pushed to the configuration proxy module through the message middleware. The configuration proxy module parses the configuration information and calls the interface to set the network configuration. The configuration proxy module sends the network configuration information of the collection port to the traffic collection module and sends the network configuration information of the forwarding port (network port group and network port information) to the VPP traffic filtering and forwarding module.

[0143] Step 3: The control agent module of the control plane sends network policy information to the specified configuration topic. The sent network policy information is pushed to the policy agent module through the message middleware. After the policy agent module parses the policy information, the policy agent module sends the application layer filtering policy to the DPI message identification module. The policy agent module sends the filtering policy and forwarding policy to the VPP high-speed traffic forwarding module. The DPI message identification module identifies the application layer protocol and tags the traffic after matching the relevant traffic to facilitate subsequent VPP policy filtering and forwarding.

[0144] Step 4. After the traffic collection module identifies the configured collection port and sets the collection method and collection port, it starts to collect data packets from the network port in polling or interrupt mode. Among them, interruption or polling can be switched based on the size of the traffic. The interrupt mechanism is used for small traffic, and polling is used for high-throughput traffic.

[0145] Step 5: Traffic is sent to the DPI module through the shared queue. After the traffic collection module forwards the traffic to the shared queue, the DPI deep packet processing module extracts the traffic from the shared queue for deep analysis, filters it based on the policy, and tags the matching packets (traffic to be forwarded).

[0146] Step 6: If DPI is not enabled, traffic is sent to the VPP module through a shared queue. The VPP filters the traffic sent by the collection module and then forwards the relevant traffic to the specified queue or network port based on the forwarding policy. The queue mode is based on same-host forwarding, and the network port mode is based on cross-host traffic forwarding.

[0147] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0148] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A scalable, high-performance network edge traffic distribution system, comprising a control plane and a data plane, each deployed in a separate central processing unit (CPU) or host. The control plane includes a configuration agent module and a policy agent module. The data plane includes a traffic collection module, a message identification module, and a traffic filtering and forwarding module. The message identification module can be enabled or disabled based on different backend integration requirements. The configuration proxy module is used to parse the network configuration information and send the network configuration information of the collection port and forwarding port to the traffic collection module and the traffic filtering and forwarding module respectively; The policy agent module is used to parse the network policy information, send the filtering policy to the message identification module, and send the filtering policy and forwarding policy to the traffic filtering and forwarding module; The traffic collection module is used to collect data packets from the network port in a polling or interrupt manner after the collection port is set based on the network configuration information of the collection port, and forward the traffic to the message identification module or the traffic filtering and forwarding module based on the start and stop status of the message identification module; wherein, Interruption or polling is switched based on the size of the traffic. The interrupt mechanism is used for small traffic, and polling is used for high-throughput traffic. The message identification module is used to extract the traffic from the shared queue for in-depth analysis after the traffic collection module forwards the traffic to the shared queue, filter the traffic based on the filtering policy, and then label the traffic; wherein the traffic is sent to the message identification module through the shared queue; The traffic filtering and forwarding module is used to filter the traffic based on the filtering policy, and then forward the traffic to a designated queue or network port according to the forwarding policy and the network configuration of the forwarding port; wherein the queue mode is based on same-host forwarding, and the network port mode is based on cross-host traffic forwarding; wherein, if the message identification module is not started, the traffic is sent to the traffic filtering and forwarding module through the shared queue.

2. The scalable, high-performance network edge traffic distribution system according to claim 1, wherein: It also includes a data plane control agent, and the data plane and the data plane control agent are arranged in the same central processor or host; The data plane control agent is used to send network configuration information and network policy information.

3. The scalable, high-performance network edge traffic distribution system of claim 1, wherein: The control plane also includes a message middleware; The message middleware is used to push network configuration information and network policy information to the configuration agent module and the policy agent module respectively.

4. The scalable, high-performance network edge traffic distribution system of claim 1, wherein: The traffic collection module starts a user state or kernel state collection engine according to the configured traffic collection mode.

5. The scalable, high-performance network edge traffic distribution system according to claim 1, wherein: The network configuration information includes the network IP, network card binding, collection port and forwarding port configuration.

6. The scalable, high-performance network edge traffic distribution system of claim 1, wherein: The message identification module adopts DPI protocol identification and separates the feature library from the engine.

7. The scalable, high-performance network edge traffic distribution system according to claim 1, wherein: The traffic filtering and forwarding module filters traffic including packet filtering, flow filtering, session filtering, application filtering and network traffic audit policy filtering.

8. The scalable, high-performance network edge traffic distribution system of claim 1, wherein: The traffic filtering and forwarding module supports Layer 2 forwarding, IP routing forwarding and multi-group combination strategies.

9. A traffic distribution method based on a scalable, high-performance network edge traffic distribution system according to any one of claims 1 to 8, characterized in that: The steps include: The configuration proxy module parses the network configuration information and sends the network configuration information of the collection port and forwarding port to the traffic collection module and traffic filtering and forwarding module respectively; The policy agent module parses the network policy information and sends the filtering policy to the message identification module, and sends the filtering policy and forwarding policy to the traffic filtering and forwarding module; After the traffic collection module sets up the collection port based on the network configuration information of the collection port, it collects data packets and forwards the traffic to the message identification module or the traffic filtering and forwarding module based on the start and stop status of the message identification module; The message identification module filters the traffic based on the filtering policy and adds a label; The traffic filtering and forwarding module filters the traffic based on the filtering policy, and then forwards the traffic according to the forwarding policy and the network configuration of the forwarding port.

Images