Extensible high-performance network boundary flow distribution system and method

Abstract

The invention relates to the technical field of network supervision, and discloses an extensible high-performance network boundary flow distribution system and method, and the method comprises the steps: a configuration agent module analyzes network configuration information, and transmits the network configuration information of a collection port and a forwarding port to a flow collection module and a flow filtering and forwarding module; the strategy agent module analyzes the network strategy information, and sends a filtering strategy and a forwarding strategy to the message identification module and the flow filtering and forwarding module; the flow acquisition module sets an acquisition port based on the network configuration information of the acquisition port, acquires a data message, and forwards the flow to the message identification module or the flow filtering and forwarding module based on the start and stop condition of the message identification module; the message identification module labels the traffic after filtering the traffic based on the filtering strategy; and the traffic filtering and forwarding module filters the traffic based on the filtering strategy and forwards the traffic according to the forwarding strategy and the network configuration of the forwarding port. And the flow forwarding fine granularity can be accurately controlled.

Description

Technical Field

[0001] The present invention relates to the technical field of network supervision, and in particular to an extensible high-performance network boundary traffic distribution system and method. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In today's information society, network security has become increasingly important. As the complexity and frequency of cyber attacks continue to increase, enterprises, institutions and individuals are facing serious threats such as data leakage, malware infection and cyber attacks. Therefore, it is necessary to protect against dangerous behaviors such as cyber attacks and to regulate the compliance of network activities.

[0004] In the process of network security prevention, control and management, due to the diversity of network traffic, understanding and processing of network traffic becomes more critical. Different types of traffic may need to go through different processing devices or different processing methods.

[0005] Therefore, pre-processing of network traffic becomes critical. In the pre-processing stage, a dedicated traffic classifier is needed to identify, filter, classify and distribute the traffic. A scalable, high-performance network edge traffic distribution system comes into being to solve complex traffic detection problems and achieve effective monitoring and security detection of large-scale network traffic.

[0006] Existing network edge traffic distribution systems are generally based on DPDK and OVS (Open vSwitch) or DPDK+VPP.

[0007] The main functions of these network edge traffic distribution systems include: Flexible and efficient traffic collection: The system supports kernel-mode and user-mode collection engines, and uses the kernel-mode AF_PACKET collection engine and the DPDK user-mode collection engine for traffic collection; Policy rule filtering and forwarding: Analyze and decide whether to allow or deny specific data packets based on the set policies, including filtering based on IP addresses, port numbers, protocol types, and session flow status. There are also systems that perform deep packet filtering based on DPI to filter dangerous and sensitive information, and forward packets based on the filtered packets. Forwarding mainly determines the forwarding path of the packet based on the destination address of the packet to ensure that the packet can reach the destination address from the source address. Efficient traffic forwarding: It realizes fast packet forwarding and processing, supports batch processing of packets, and efficiently handles large amounts of concurrent traffic, thereby improving overall system throughput and providing efficient distribution capabilities for back-end business processing.

[0008] These network edge traffic distribution systems generally have the following problems: (1) Performance bottleneck: Low performance is the main disadvantage of software forwarding. If it is not based on hardware conditions, forwarding may become a bottleneck in a high-traffic environment, resulting in increased latency and decreased throughput, especially when processing a large amount of data packets.

[0009] (2) Resource consumption: The granularity of filtering is relatively coarse. In fact, traffic forwarding requires the consumption of CPU and memory resources, especially when a large number of data packets need to be processed, which may lead to excessive occupation of system resources. In high-traffic scenarios, higher CPU and memory resources may be required. In particular, when performing deep packet processing, resource consumption will increase significantly.

[0010] (3) Poor scalability: These forwarding systems are generally completely separated from the business system, do not uniformly manage business policies, and have relatively scattered business targeting. At the same time, these systems are generally deployed on a single device for business modules. For large-scale networks, especially those involving high throughput and large-scale concurrent connections, the traffic forwarding of such systems often lacks good scalability. Summary of the invention

[0011] In order to solve the above problems, the present invention provides a scalable high-performance network boundary traffic distribution system and method to solve the pre-filtering and distribution problems of traffic. According to different back-end integration requirements, the DPI-related functional modules can be started and stopped to accurately control the fine-grained traffic forwarding, thereby providing support for subsequent business to process large traffic.

[0012] In order to achieve the above object, the present invention adopts the following technical solution: The first aspect of the present invention provides an extensible high-performance network edge traffic distribution system, which includes a control plane and a data plane arranged in different central processors or hosts; the control plane includes a configuration agent module and a policy agent module; the data plane includes a traffic collection module, a message identification module and a traffic filtering and forwarding module, and the message identification module is started and stopped according to different back-end integration requirements; The configuration proxy module is used to parse the network configuration information and send the network configuration information of the collection port and the forwarding port to the flow collection module and the flow filtering and forwarding module respectively; The policy proxy module is used to parse the network policy information, and send the filtering policy to the message identification module, and send the filtering policy and forwarding policy to the traffic filtering and forwarding module; The traffic collection module is used to collect data messages after the collection port is set based on the network configuration information of the collection port, and forward the traffic to the message identification module or the traffic filtering and forwarding module based on the start and stop status of the message identification module; The message identification module is used to filter the traffic based on the filtering policy and then add a label; The traffic filtering and forwarding module is used to filter the traffic based on the filtering policy, and then forward the traffic according to the forwarding policy and the network configuration of the forwarding port.

[0013] Furthermore, it also includes a data plane control agent, and the data plane and the data plane control agent are arranged in the same central processor or host; The data plane control agent is used to send down network configuration information and network policy information.

[0014] Furthermore, the control plane also includes a message middleware; The message middleware is used to push network configuration information and network policy information to the configuration agent module and the policy agent module respectively.

[0015] Furthermore, the traffic collection module collects traffic from the network port in a polling or interruption manner.

[0016] Furthermore, the traffic collection module starts a user state or kernel state collection engine according to the configured traffic collection mode, and switches to a polling or interrupt mode to collect traffic according to the size of the traffic.

[0017] Furthermore, the network configuration information includes the configuration of network IP, network card binding, collection port and forwarding port.

[0018] Furthermore, the message identification module adopts DPI protocol identification and separates the feature library from the engine.

[0019] Furthermore, the traffic filtering and forwarding module filters the traffic including packet filtering, stream filtering, session filtering, application filtering and network traffic audit policy filtering; Furthermore, the traffic filtering and forwarding module supports Layer 2 forwarding, IP routing forwarding and multi-group combination strategies.

[0020] A second aspect of the present invention provides a traffic distribution method of a scalable high-performance network edge traffic distribution system based on the first aspect, comprising the following steps: The configuration proxy module parses the network configuration information and sends the network configuration information of the collection port and the forwarding port to the traffic collection module and the traffic filtering and forwarding module respectively; The policy proxy module parses the network policy information, sends the filtering policy to the message identification module, and sends the filtering policy and forwarding policy to the traffic filtering and forwarding module; After the traffic collection module sets the collection port based on the network configuration information of the collection port, it collects data packets and forwards the traffic to the packet identification module or the traffic filtering and forwarding module based on the start and stop status of the packet identification module; The message identification module filters the traffic based on the filtering policy and adds a label; The traffic filtering and forwarding module filters the traffic based on the filtering policy, and then forwards the traffic according to the forwarding policy and the network configuration of the forwarding port.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a scalable, high-performance network edge traffic distribution system, which solves the problems of pre-filtering and distribution of traffic. According to different back-end integration requirements, it can start and stop DPI-related functional modules to accurately control the fine-grained traffic forwarding, thereby providing support for subsequent business to process large traffic.

[0022] The present invention provides a scalable high-performance network edge traffic distribution system, which performs traffic filtering and forwarding control through configuration and strategy. In specific scenarios, full traffic forwarding is required. The control plane can control the component functions of the data plane and start different module functions as needed.

[0023] The present invention provides a scalable, high-performance network edge traffic distribution system, which applies DPDK technology in the traffic collection part, optimizes the packet receiving architecture, and maximizes the system traffic collection performance; VPP technology is applied in the data plane traffic forwarding part, and the function expansion based on VPP is performed to greatly improve the data packet filtering and forwarding capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute limitations of the present invention.

[0025] Figure 1 This is a schematic diagram of separation of the control plane and the data plane in Embodiment 1 of the present invention; Figure 2 This is a flow chart of the flow collection process of the first embodiment of the present invention; Figure 3 This is a schematic diagram of the front flow filtering strategy control of the first embodiment of the present invention; Figure 4 This is a DPI data processing flow chart of the first embodiment of the present invention; Figure 5This is a schematic diagram of the VPP forwarding plane pipeline processing according to the first embodiment of the present invention; Figure 6 This is a logical diagram of a traffic distribution framework according to the first embodiment of the present invention; Figure 7 This is an overall architecture diagram of a network boundary traffic detection framework according to the first embodiment of the present invention; Figure 8 This is a processing sequence diagram of the system according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0028] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0029] In the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention is further described below with reference to the accompanying drawings and embodiments.

[0030] Embodiment 1 The purpose of this embodiment 1 is to provide a scalable, high-performance network edge traffic distribution system.

[0031] This embodiment provides a scalable high-performance network edge traffic distribution system that can provide efficient and accurate fine-grained control and forwarding of network traffic and filter unnecessary network traffic. The main features include separation of the control plane and the data plane, a flexible traffic collection engine, policy rule filtering and policy management, high-performance traffic forwarding, and system monitoring and management.

[0032] This embodiment provides a scalable high-performance network edge traffic distribution system, which mainly has the following functions: Flexible and efficient traffic collection: The system supports kernel-mode and user-mode collection engines, using the traditional kernel-mode AF_PACKET collection engine and the DPDK user-mode collection engine for traffic collection. The traffic collection mode can be switched through the API interface.

[0033] Unified network configuration management: Centrally manage the network configuration of different components and sub-services to achieve docking coordination and resource consistency in the same host and cross-host network environment.

[0034] Policy rule filtering and forwarding: Based on the multi-dimensional traffic access control list mechanism, a multi-dimensional traffic filtering capability is formed, supporting event-level control, black and white list control, application-level control, session-level control and user-level control; the rule filtering engine implements layer-by-layer unloading processing of collected traffic, such as packet filtering, flow filtering, session filtering, application filtering, and network traffic audit policy filtering, and through rich filtering rules, hierarchical traffic filtering and unloading acceleration processing.

[0035] Front-end DPI deep packet analysis: Through DPI deep packet analysis, intelligent traffic control is performed, DPI message analysis is coordinated with security forwarding strategies to optimize backend traffic quality, filter out unnecessary traffic messages, and provide session-level and application-level traffic control capabilities.

[0036] VPP high-performance data processing plane: It realizes fast packet forwarding and processing, supports batch processing of packets, and efficiently handles a large amount of concurrent traffic, thereby improving the overall system throughput and providing efficient distribution capabilities for back-end business processing.

[0037] This embodiment provides a scalable high-performance network edge traffic distribution system, such as Figure 1 As shown, it includes a control plane, a data plane, and a data plane control agent (also known as a control agent module). The control plane and the data plane are set in different central processors or hosts, and the data plane and the data plane control agent are set in the same central processor or host. The control plane is separated from the data plane to reduce functional coupling, independent maintenance, reduce complexity, reduce maintenance requirements, and reduce the scope of the fault domain.

[0038] Among them, the control plane is responsible for system initialization, configuration changes, policy updates, and data queries.

[0039] Among them, the data plane is responsible for uninterrupted data capture and analysis.

[0040] The control plane and data plane can exist and upgrade independently; the control plane and data plane can run independently and be flexibly deployed.

[0041] Among them, Figure 7 As shown, the control plane ZNMS manages and issues configuration, policies and commands. The control plane includes message middleware, configuration agent module and policy agent module.

[0042] Among them, ZNMS uses Kafka as the message middleware to send relevant control and business data to the traffic collection and forwarding module. Among them, the strategies include black and white IP list filtering and forwarding, IP filtering, port filtering, and protocol filtering; the control commands include host start and stop, module start and stop, sub-service status monitoring, etc.

[0043] Among them, the policy proxy module ZNDC-Sub-Server (also known as the API policy receiving module) receives the policy from the message middleware and parses and forwards it. The policy proxy module mainly parses the specific policy issued by the management module, reassembles the policy according to the module number issued by the policy, and forwards it to the corresponding module or sub-service. Functional control and interface settings are carried out through policies, including specific policy actions.

[0044] Among them, the configuration agent module (also known as the configuration management module) can uniformly configure the forwarding framework according to specific business needs, and the configuration management module can uniformly control the operation of the entire business system and monitor the business operation status, data processing format, etc.

[0045] The data plane includes traffic collection and forwarding modules. The traffic collection and forwarding module ZNDC integrates the DPDK packet receiving module (also known as the traffic collection module), the DPI deep packet inspection module (also known as the message identification module) and the VPP high-speed traffic forwarding module (also known as the traffic filtering and forwarding module), supports cross-host network port forwarding and zero-copy shared queue distribution on the same host, and also supports virtual network port traffic forwarding on the same host, such as Figure 6 shown.

[0046] The traffic collection module uses a flexible traffic collection engine: it is used for collecting, sharing, ACL forwarding, etc. of network traffic; it achieves high-performance packet capture through zero-copy technology, and can receive real-time network traffic, replay traffic, or directly read PCAP files.

[0047] Among them, Figure 2 As shown, the traffic collection module is configured as follows: configure the traffic collection mode; start the user state or kernel state collection engine according to the configured traffic collection mode; flexibly switch events or polling mechanisms to collect traffic based on the traffic size; collect all traffic and obtain traffic data packets from the network card queue; send the traffic to the processing queue for subsequent BPI analysis, policy matching and forwarding.

[0048] Among them, the traffic collection module can control the data access method (traffic collection mode) through configuration, including DPDK traffic collection, PF_RING traffic collection, AF_PACKET traffic collection and LIBPCAP library-based traffic collection, and intelligently set relevant configurations according to specific business performance requirements.

[0049] Among them, the message identification module adopts DPI protocol identification. The DPI message identification module adopts the method of separating the feature library and engine processing, making the feature library independent of the system engine; the feature library compilation module is responsible for analyzing application features and compiling the features into feature library files; the main engine processing module is responsible for loading and parsing the feature library files, and performing data packet matching processing at the same time.

[0050] Among them, the data message parsing of the message identification module supports L2-L7 layer protocol identification and matching, and can filter traffic based on IP, port, application layer protocol, application name, session flow, etc.

[0051] Currently, the DPI layer supports the processing of TCP / UDP traffic, and the data processing flow of the message identification module is as follows: Figure 4As shown, it includes: (1) TCP / UDP judgment: check whether the traffic is TCP or UDP protocol; if not (N), go to step (12); if yes (Y), go to step (2). (2) Hash table query: perform hash table query from the flow through the five-tuple (source IP, source port, destination IP, destination port, protocol). If the query is found, go to step (3); if not, go to step (4). (3) Query the mark in the flow to determine whether the flow is terminated; if not, go to step (10); if it is terminated, determine whether there is an identification result. If there is an identification result, determine whether the maximum number of processing is reached. If the maximum number of processing is reached, go to step (10); if there is no identification result or the maximum number of processing is not reached, go to step (5). (4) Create a new flow and enter the matching process. If there is a blacklist before matching, perform blacklist matching first and determine whether the blacklist filtering is successful. If successful (Y), the process ends; if unsuccessful (N), go to step (5). (5) Pre-filter matching: Pre-filter matching is performed by setting the filter hash table query; if the match is successful (Y), go to step (10); if the match is unsuccessful (N), go to step (6). (6) Pre-grouping: Pre-group the traffic, including HTTP groups, TLS groups, TCP groups and UDP groups, to speed up the matching speed. (7) HS regular matching: Perform HS regular matching and return the regular matching rule ID and the matching content location. (8) High-speed pattern matching module matching: Through the rule ID result matched by the high-speed pattern matching module, reverse search the corresponding protocol classifier, and then match according to the protocol classifier. (9) Protocol information extraction: Extract protocol information from the identified traffic. (10) Post-processing: Post-process the traffic that has been identified or not identified but has reached the maximum depth. (11) IANA protocol matching, DNS setting filter, IP blacklist setting: After post-processing, perform IANA protocol matching, DNS setting filter and IP blacklist setting. (12) Forwarding: Finally, the processed traffic is forwarded.

[0052] The traffic filtering and forwarding module is configured with a pre-placed policy rule filtering submodule and a traffic forwarding control submodule.

[0053] like Figure 3 As shown in the figure, the policy rule filtering submodule performs layer-by-layer offloading processing such as packet filtering, flow filtering, session filtering, application filtering, and network traffic audit policy filtering on the collected traffic. Through rich filtering rules, hierarchical traffic filtering, and offload acceleration processing design, it can effectively reduce the complexity of protocol depth analysis and calculation, and can accurately and effectively filter unnecessary traffic and screen out the traffic messages required by the business system.

[0054] The traffic forwarding control submodule (also known as the traffic control and forwarding module) supports Layer 2 forwarding, including Layer 2 forwarding capabilities for VLAN and QinQ packets; supports IP routing forwarding, including support for IPv4 and IPv6; supports multi-group combination strategies, including but not limited to IP blacklists and whitelists, ports, protocols, etc., and can perform operations such as packet filtering, distribution, and redirection.

[0055] Among them, the traffic forwarding control submodule supports traffic distribution between processes across hosts and within the same host. The same host is mainly for single device resource scenarios, integrating back-end services into the same business system; cross-host is only for scenarios with high-traffic business scenarios where the resources of a single device are limited and the processing performance cannot meet actual needs.

[0056] In order to support flexible traffic filtering functions, support multi-dimensional traffic filtering strategies, and improve the processing performance of traffic collection and forwarding modules, the next step of distribution processing is only carried out when the strategy matches and the action is Permit.

[0057] The forwarding plane pipeline processing of VPP is as follows Figure 5 As shown, VPP modifies and expands the original pipeline: (1) Use ACL plugin as the basic ACL policy control, but need to extend ACL action to associate action profile description configuration; and add session node / DPI node / distribution node / IP fragment pre-reassembly node. The newly added node is inserted by dynamic control arrangement using API; (2) Distribution policy control based on ACL+DPI allows specific 5-tuple (Layer 3 protocol) messages + specific applications / Layer 4 protocols / URLs to be distributed to specific targets for further security analysis and processing; (3) The distribution policy can be configured to support 8-tuples, 4-layer port association, and ACL rules for IP fragment aggregation search with the same SRCIP+DSTIP+IPID (fragment ID). It can also implement packet aggregation search, aggregation processing, and forwarding through IP fragment pre-reassembly. (4) In vlib_buffer_t, construct the opaque3 context information field. The opaque3 field is used to transmit context information between nodes in the arc chain. The main information includes: the five-tuple information corresponding to the packet, the policy rule number and the associated action profileid, the policy application ID, the flag bit that needs to be transmitted between nodes, the NAT information, etc. (5) The distribution node is responsible for target redirection distribution. It performs secure application redirection distribution according to the configured load sharing strategy, target list, and NAT strategy, and the distribution action profile description configuration; (6) The session node accelerates the subsequent data of the same flow by recording the five-tuple of each flow, as well as the policy rule number corresponding to the flow, the associated action profileid, the policy application ID, the processing action of the subsequent node, and other information. When the NAT rule is issued, the session node can also record the NAT reverse session; (7) The policy node / DPI node / distribution node provides the corresponding binary API interface to the upper layer for configuration; (8) IP6 flow is similar to IP4.

[0058] VPP forwarding plane pipeline processing Figure 5 As shown, where: Packet vector: This is the input source of the data packet, which contains multiple input interfaces, such as P1, P2, P3, etc. These interfaces may represent different network interface cards (NICs).

[0059] Starting from the Packet vector, there are multiple input paths connected to different processing nodes. Processing nodes and their functions: af - packet - input: Receives packets from Packet vector, which may be input based on the AF_PACKET mechanism; mpls - input: handles MPLS (Multi-Protocol Label Switching) packets; ethernet-input: processes Ethernet packets; dpdk - input: Receive packets using DPDK (Data Plane Development Kit), typically used for high-performance packet processing; l2 - input: processes layer 2 (data link layer) packets; l2-dispatch: distributes and processes layer 2 data packets; ip6 - input: processes IPv6 packets; llc - input: handles LLC (Logical Link Control) packets; pppoe - input: handles PPPoE (Point-to-Point Protocol over Ethernet) packets; arp - input: Processes ARP (Address Resolution Protocol) packets.

[0060] Core processing path: ip4 - input: processes IPv4 packets; ip4 - sv - reassembly - policy: Perform segmentation and reassembly policy processing on IPv4 packets; ip4 - policy - input: Apply IPv4 policy input; ip4 - zdpi - handoff: passes the packet to the ZDPI (Zero Day Packet Detection and Prevention) module; ip4 - zdpi: performs ZDPI processing; ip4 - zdpi - gtpu: handles GTP-U (GPRS Tunneling Protocol - User Plane) packets; ip4 - zdpi - capwap: handles CAPWAP (Control and Configuration of Wireless Access Points Protocol) packets; ip4 - lookup: perform lookup operations on IPv4 packets; ip4 - cat - x: performs some sort of classification operation (perhaps based on categories or tags); ip4-dispatch: distribute IPv4 packets; ip4-buffer-duplication: duplicate IPv4 packet buffer; ip4 - fa: may be some kind of forwarding action; ip4 - cat - snat - static: performs static source NAT (Network Address Translation) operation; ip4 - rewrite: rewrite IPv4 packets; ip4 - policy - output: applies IPv4 policy output; zach - plugin - in - ip4 - fa: Apply a plugin (possibly the Zach plugin) to perform IPv4 forwarding operations; zach - plugin - out - ip4 - fa: the output after applying a plugin to perform IPv4 forwarding operations; ip4 - firewall: perform IPv4 firewall operations; nat44_session_handoff: handles NAT44 (Network Address Translation version 44) session handoff; Interface - output: The processed data packets are finally output to the network interface.

[0061] like Figure 8 As shown in the figure, the data flow steps between the control proxy module, message middleware, configuration proxy module, policy proxy module, DPDK packet receiving module (also known as traffic collection module), DPI deep packet inspection module (also known as message identification module) and VPP high-speed traffic forwarding module (also known as traffic filtering and forwarding module) include: The control agent module (data plane control agent) of the control plane sends network configuration information to the specified configuration Topic, including basic network IP, network card binding, collection port and forwarding port (network port group, network port) configuration. The sent network configuration information is pushed to the configuration agent module through the message middleware. The configuration agent module parses the configuration information, calls the interface to set the network configuration, and sends the network configuration information of the collection port to the traffic collection module, and sends the network configuration information of the forwarding port (network port group and network port configuration information) to the VPP traffic filtering and forwarding module. The control agent module of the control plane sends network policy information to the specified configuration Topic; the sent network policy information is pushed to the policy agent module through the message middleware. After the policy agent module parses the policy information, the policy agent module sends the application layer filtering policy to the DPI message identification module. The policy agent module sends the filtering policy and forwarding policy to the VPP high-speed traffic forwarding module. The DPI message identification module identifies the application layer protocol and tags the traffic after matching the relevant traffic, which facilitates the subsequent VPP policy filtering and forwarding. After the traffic collection module identifies the configured collection port and sets the collection mode and collection port based on the network configuration information of the collection port, it starts to collect data packets from the network port in a polling or interruption manner, and forwards the traffic to the DPI message identification module or the VPP traffic filtering and forwarding module based on the start and stop status of the message identification module; among which, the interruption or polling can be switched based on the size of the traffic, and the interruption mechanism is used for small traffic, and the polling mechanism is used for high-throughput traffic. The traffic is sent to the DPI module through a shared queue. The DPI deep message processing module takes out the traffic from the shared queue for deep analysis after the traffic collection module forwards the traffic to the shared queue, filters the traffic based on the filtering policy, and tags the matching messages (traffic to be forwarded). If DPI is not started, the traffic is sent to the VPP module through a shared queue. The VPP filters the traffic sent by the collection module according to the policy, and then forwards the relevant traffic to the specified queue or network port according to the forwarding policy and the network configuration of the forwarding port; among which, the queue mode is based on same-host forwarding, and the network port mode is based on cross-host traffic forwarding. This embodiment provides a scalable high-performance network edge traffic distribution system. In terms of hardware-level resource optimization, DPDK technology is used to optimize the packet receiving architecture, provide an interface to configure hardware resources, and ensure that the system obtains appropriate hardware resources at the packet receiving level. In addition, through the resource partitioning competition elimination method, as hardware resources increase, the system's performance scalability is also improved, and it can support 20Gbps network traffic processing.

[0062] This embodiment provides a scalable, high-performance network edge traffic distribution system that adopts flexible policy control and supports pre- and post-policy control. In the traffic collection and distribution processing module, policies can be used to flexibly control traffic filtering and forwarding channels. In the traffic parsing and analysis module, policies can be used to control the granularity of traffic parsing and analysis.

[0063] This embodiment provides a scalable high-performance network edge traffic distribution system, which uses a load balancing mechanism to distribute traffic detection tasks to ensure uniform load among distribution nodes.

[0064] This embodiment provides a scalable high-performance network edge traffic distribution system that supports horizontal and vertical expansion and can dynamically increase or decrease nodes as needed to cope with traffic fluctuations.

[0065] This embodiment provides a scalable high-performance network edge traffic distribution system that supports plug-in detection algorithms and analysis tools, making it easy to integrate new detection technologies and algorithms.

[0066] This embodiment provides a scalable, high-performance network edge traffic distribution system, which mainly solves the problem of pre-filtering and distribution of traffic. According to different business scenarios, it accurately controls the fine-grained granularity of traffic forwarding to provide support for subsequent business systems to process large traffic.

[0067] This embodiment provides a scalable, high-performance network edge traffic distribution system. Since traffic filtering and forwarding control are performed through configuration and policies, full traffic forwarding is required in specific scenarios. It is necessary to control the component functions of the data plane on the control plane and start different module functions as needed.

[0068] This embodiment provides a scalable, high-performance network edge traffic distribution system, which applies DPDK technology in the traffic collection part, optimizes the packet receiving architecture, and maximizes the system traffic collection performance; VPP technology is applied in the data plane traffic forwarding part, and the function expansion based on VPP greatly improves the data packet filtering and forwarding capabilities.

[0069] This embodiment provides a scalable, high-performance network edge traffic distribution system, which can start and stop DPI-related functional modules according to different back-end integration requirements, and only provide five-tuple traffic filtering functions to improve overall processing performance; it adopts a modular design and can flexibly adjust the deployment method.

[0070] This embodiment provides a scalable, high-performance network edge traffic distribution system. To reduce the difficulty of deployment and maintenance management, the overall filtering and distribution framework provides a unified operation and maintenance management API and a visualization interface, and function configuration and module start and stop are performed through the API or visualization page.

[0071] Embodiment 2 The purpose of this second embodiment is to provide a traffic distribution method of a scalable high-performance network edge traffic distribution system based on the first embodiment, including the following steps: Step 1: Send network configuration information to the specified configuration Topic through the control agent module of the control plane, including basic network IP, network card binding, network port group and forwarding port configuration.

[0072] Step 2: The issued network configuration information is pushed to the configuration proxy module through the message middleware. The configuration proxy module parses the configuration information and calls the interface to set the network configuration. The configuration proxy module sends the network configuration information of the collection port to the traffic collection module and sends the network configuration information of the forwarding port (network port group and network port information) to the VPP traffic filtering and forwarding module.

[0073] Step 3: The control agent module of the control plane sends the network policy information to the specified configuration Topic. The sent network policy information is pushed to the policy agent module through the message middleware. After the policy agent module parses the policy information, the policy agent module sends the application layer filtering policy to the DPI message identification module. The policy agent module sends the filtering policy and forwarding policy to the VPP high-speed traffic forwarding module. The DPI message identification module identifies the application layer protocol and tags the traffic after matching the relevant traffic, which facilitates the subsequent VPP policy filtering and forwarding.

[0074] Step 4: After the traffic collection module identifies the configured collection port and sets the collection method and collection port, it starts to collect data packets from the network port in polling or interrupt mode; wherein, interruption or polling can be switched based on the size of the traffic, with an interrupt mechanism used for small traffic and polling for high-throughput traffic.

[0075] Step 5: Traffic is sent to the DPI module through the shared queue. After the traffic collection module forwards the traffic to the shared queue, the DPI deep packet processing module takes out the traffic from the shared queue for deep analysis, filters it based on the policy, and tags the matching packets (traffic to be forwarded).

[0076] Step 6: If DPI is not started, the traffic is sent to the VPP module through the shared queue. VPP filters the traffic sent by the collection module according to the policy, and then forwards the relevant traffic to the specified queue or network port according to the forwarding policy. Among them, the queue mode is based on same-host forwarding, and the network port mode is based on cross-host traffic forwarding.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0078] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An extensible high-performance network edge traffic distribution system, comprising a control plane and a data plane arranged in different central processors or hosts; the control plane comprises a configuration proxy module and a policy proxy module; the data plane comprises a traffic collection module, a message identification module and a traffic filtering and forwarding module, and the message identification module is started and stopped according to different backend integration requirements; The configuration proxy module is used to parse the network configuration information and send the network configuration information of the collection port and the forwarding port to the flow collection module and the flow filtering and forwarding module respectively; The policy proxy module is used to parse the network policy information, and send the filtering policy to the message identification module, and send the filtering policy and forwarding policy to the traffic filtering and forwarding module; The traffic collection module is used to collect data messages after the collection port is set based on the network configuration information of the collection port, and forward the traffic to the message identification module or the traffic filtering and forwarding module based on the start and stop status of the message identification module; The message identification module is used to filter the traffic based on the filtering policy and then add a label; The traffic filtering and forwarding module is used to filter the traffic based on the filtering policy, and then forward the traffic according to the forwarding policy and the network configuration of the forwarding port.

2. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: It also includes a data plane control agent, and the data plane and the data plane control agent are arranged in the same central processor or host; The data plane control agent is used to send down network configuration information and network policy information.

3. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: The control plane also includes a message middleware; The message middleware is used to push network configuration information and network policy information to the configuration agent module and the policy agent module respectively.

4. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: The flow collection module collects flow from the network port in a polling or interruption manner.

5. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: The flow collection module starts a user state or kernel state collection engine according to the configured flow collection mode, and switches to polling or interrupt mode to collect flow according to the size of the flow.

6. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: The network configuration information includes the configuration of network IP, network card binding, collection port and forwarding port.

7. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: The message identification module adopts DPI protocol identification and separates the feature library from the engine.

8. The scalable high-performance network edge traffic distribution system according to claim 1, characterized in that: The traffic filtering and forwarding module filters the traffic including packet filtering, stream filtering, session filtering, application filtering and network traffic audit policy filtering.

9. A scalable high-performance network edge traffic distribution system as claimed in claim 1, characterized in that: The traffic filtering and forwarding module supports layer 2 forwarding, IP routing forwarding and multi-group combination strategies.

10. A traffic distribution method of a scalable high-performance network edge traffic distribution system according to any one of claims 1 to 9, characterized in that: The steps include: The configuration proxy module parses the network configuration information and sends the network configuration information of the collection port and the forwarding port to the traffic collection module and the traffic filtering and forwarding module respectively; The policy proxy module parses the network policy information, sends the filtering policy to the message identification module, and sends the filtering policy and forwarding policy to the traffic filtering and forwarding module; After the traffic collection module sets the collection port based on the network configuration information of the collection port, it collects data packets and forwards the traffic to the packet identification module or the traffic filtering and forwarding module based on the start and stop status of the packet identification module; The message identification module filters the traffic based on the filtering policy and adds a label; The traffic filtering and forwarding module filters the traffic based on the filtering policy, and then forwards the traffic according to the forwarding policy and the network configuration of the forwarding port.

Images