A method and system for identifying and parsing large bandwidth network traffic protocols

Abstract

The present invention discloses a method and system for identifying and parsing large-bandwidth network traffic protocols, which relates to the technical field of network traffic protocol parsing and processing. The method includes: receiving the original traffic of the network and storing it in the buffer in the kernel state in the form of data frames, based on the zero-copy technology, directly mapping the data frames in the kernel-state buffer to the user-state memory area, logically classifying the data stream and sending it to the corresponding computing processing cores through a dynamic distribution mechanism, and each of the computing processing cores completes the layer-by-layer identification and parsing of the traffic protocol and extracts the corresponding metadata from the protocol identification result. The present invention adopts a protocol identification method with a three-level pipelined architecture in the computing processing core to complete the layer-by-layer identification and parsing of the traffic protocol, avoiding high-complexity in-depth parsing of all traffic. When facing new or complex protocols, there is no need to reconstruct the entire identification process, and each stage can be independently optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic protocol parsing and processing, and specifically to a method and system for identifying and parsing large-bandwidth network traffic protocols. Background Art

[0002] Existing protocol identification and parsing methods mainly include feature matching and deep packet inspection (DPI). Among them, the feature matching method identifies by analyzing specific fields or key features of the protocol (such as port numbers, byte sequences). However, when dealing with complex protocols and newly added protocols in large-bandwidth traffic, the accuracy of protocol identification is relatively low, and the maintenance cost for newly added protocols is relatively high. DPI identifies protocols by deeply parsing the content of data packets. Although the accuracy is high, the computational complexity is large, especially when facing large-bandwidth traffic, it is difficult to meet the real-time requirements. The above methods have deficiencies in terms of real-time performance, scalability, and resource utilization when dealing with complex bandwidth network traffic. Summary of the Invention

[0003] I. Technical Problems to be Solved

[0004] The present invention provides a method and system for identifying and parsing large-bandwidth network traffic protocols to solve the deficiencies of existing protocol identification and parsing methods in terms of real-time performance, scalability, and resource utilization when dealing with complex bandwidth network traffic.

[0005] II. Technical Solutions

[0006] To achieve the above object, the present invention provides the following technical solutions: A method for identifying and parsing large-bandwidth network traffic protocols, comprising the following steps:

[0007] Receive the original traffic of the network and store it in the buffer in the kernel state in the form of data frames;

[0008] Based on the zero-copy technology, directly map the data frames in the kernel state buffer to the user state memory area as the starting point for the data stream to enter the user state for processing;

[0009] According to the principle of the same source and the same destination, logically classify the data stream with the source address, destination address, source port, and destination port as the classification basis, and send the classified data stream to the corresponding computing processing cores through a dynamic distribution mechanism; wherein, the dynamic distribution mechanism distributes the data stream to different computing processing cores through a load balancing strategy;

[0010] Each of the computing and processing cores completes layer-by-layer parsing of the allocated data stream through a three-stage pipelined architecture. Among them, the three-stage pipelined architecture includes three parsing stages: underlying protocol identification, pattern matching identification, and advanced protocol matching identification. The underlying protocol identification determines the underlying protocol type by parsing link layer and network layer fields. The pattern matching identification parses the middle-level protocol through field matching rules and a predefined pattern library. The advanced protocol matching identification parses complex protocols by combining context information and a state machine mechanism. After the data stream completes parsing in each stage, if the protocol type is determined, the result is directly output; otherwise, it enters the next stage for parsing.

[0011] Based on the protocol identification result, an adapted metadata extraction model is called in each of the computing and processing cores to extract metadata from the identification result. Among them, each metadata extraction model is bound to protocol characteristics and is automatically matched and called according to the result of the protocol parsing stage.

[0012] Each of the computing and processing cores combines the protocol identification result and the metadata extraction result into a data queue and then outputs it.

[0013] Furthermore, the zero-copy technology is specifically the MFRU header data prefetch management algorithm based on zero-copy. This algorithm adopts a two-level table structure of a first-level free list stack and a second-level doubly linked free list, and allocates a free list entry in the kernel-mode buffer to store the header data in the data frame. Specifically:

[0014] When the number of free list entries in the buffer is lower than a set first set threshold, the algorithm invents page faults to the first-level free list stack to expand the number of available list entries.

[0015] When the number of free list entries in the buffer is greater than a set second set threshold, the algorithm releases the free list entries and reclaims them into the first-level free list stack.

[0016] Furthermore, the load balancing strategy is specifically to adopt a feedback-based two-level balanced routing algorithm for two-level distribution of the data stream. Specifically:

[0017] The first level obtains the real-time load information of each computing node and sends the data stream to the computing nodes according to the load situation based on the principle of the same source and the same destination.

[0018] The second level dynamically adjusts the hash algorithm inside each computing node based on the load monitoring data of the computing and processing cores and then distributes the data stream to each computing and processing core again to achieve balanced distribution of the data stream within the computing node.

[0019] Further, during the identification and parsing process of the protocol, a distributed hash flow table management algorithm based on the Splay tree is used to construct a distributed flow table according to the number of the computing and processing cores. Each computing and processing core independently maintains a flow table with a Splay tree structure. When the data stream arrives at the computing and processing core, the corresponding flow table entry will be searched in the Splay tree;

[0020] If the corresponding flow table entry is found, the protocol status stored in the flow table will be directly obtained;

[0021] If the corresponding flow table entry is not found, a new flow table entry will be created and inserted into the Splay tree.

[0022] Further, the underlying protocol identification and parsing stage in the three-level pipeline architecture is responsible for quickly identifying the link layer and network layer protocol types of the data packet, which serves as the basic stage of protocol parsing. Specifically, in this stage, preliminary identification is carried out based on the basic characteristics of the data packet and the protocol header information to identify the data link layer and the network layer;

[0023] If the link layer or network layer protocol type is successfully identified, the protocol identification result will be directly output and the data stream parsing will be completed;

[0024] If the protocol type cannot be determined, the data stream will enter the pattern matching identification and parsing stage of the next stage.

[0025] Further, the pattern matching identification and parsing stage in the three-level pipeline architecture is responsible for further identifying the transport layer protocol type of the data packet. By comparing the data packet content with the predefined matching patterns, the transport layer protocol type is confirmed. Specifically, a minimum perfect hash function is constructed to map the predefined matching patterns to unique hash values to avoid hash conflicts and achieve fast matching. Regular expression matching and signature matching are used to locate the protocol feature fields;

[0026] If the transport layer protocol type is successfully confirmed, the protocol identification result will be directly output and the data stream parsing will be completed;

[0027] If the protocol type cannot be determined, the data stream will enter the advanced protocol matching of the next stage.

[0028] Further, the advanced protocol matching and identification parsing stage in the three-level pipeline architecture is responsible for identifying complex application layer protocol types. This stage combines the logical association data packet identification method and the statistical analysis feature method to identify the application layer protocol; correspondingly:

[0029] The logical association data packet identification method defines the protocol context by analyzing the logical association relationship between data packets;

[0030] The multi-association layer matching method judges the application layer protocol type by integrating the information and association relationship of multiple data packets;

[0031] The statistical analysis feature method analyzes the protocol by using traffic statistical features;

[0032] Output the complete protocol parsing result to complete the parsing of the data stream.

[0033] Furthermore, based on the protocol recognition result, an adapted metadata extraction model is called in each of the computing processing cores to extract metadata. Correspondingly:

[0034] The packet-driven metadata extraction model extracts relevant metadata after extracting the IP header and data part in a single data packet;

[0035] The flow-driven metadata extraction model extracts relevant metadata after analyzing the data packets in the entire data stream;

[0036] The tunnel logical session-driven metadata extraction model extracts logical session information from the tunnel protocol data and extracts relevant metadata after analyzing the logical session data;

[0037] The event-driven metadata extraction model first extracts the IP header and data part in the data, and extracts the corresponding metadata after analysis according to the protocol requirements.

[0038] Furthermore, each of the computing processing cores integrates the protocol recognition result and the metadata extraction result into a data queue, and hands over the data queue to be managed by a double-ended lock-free linked list. Specifically:

[0039] When the output of the data queue is completed, it is marked as processed, and the computing processing core stores the processed data queue into the double-ended lock-free linked list;

[0040] Recycle the packet header table entries in the data queue through the MFRU packet header data prefetch management algorithm, and re-add them to the first-level free list stack for subsequent data streams to use.

[0041] A parsing system for a large-bandwidth network traffic protocol recognition method is used to implement any of the above-mentioned protocol recognition and parsing methods. The parsing system adopts an architecture based on the separation of data, control, and computing planes, where:

[0042] The data plane is used to receive network traffic data, perform preliminary classification and distribution on the data to ensure that the traffic can enter the user state for processing; among them, the reception, classification, and distribution of traffic data are completed through the MFRU packet header data prefetch management algorithm and the second-level feedback-based equalization routing algorithm;

[0043] The computing plane is used for protocol identification and parsing of data streams and metadata extraction; among them, the traffic protocol identification and parsing and metadata extraction are completed through the Splay tree flow table management algorithm, and the data queue is managed in combination with the MFRU packet header data prefetch management algorithm;

[0044] The control plane is used to manage resource allocation, policy adjustment, and global load balancing; among them, the flow table optimization and global load balancing are performed through the Splay tree flow table management algorithm and the two-level feedback equalization routing algorithm.

[0045] (3) Beneficial effects:

[0046] Compared with the prior art, the invention has the following beneficial effects:

[0047] The present invention dynamically adjusts routing distribution based on traffic load to ensure uniform distribution of the load of each computing and processing core in the multi-core processor architecture.

[0048] The present invention adopts a three-stage pipelined architecture protocol identification method in the computing and processing core, that is, the data stream distributed into the processing core sequentially passes through the underlying protocol identification, pattern matching identification, and advanced protocol matching to complete the layer-by-layer identification and parsing of the traffic protocol; the three-stage pipelined architecture analyzes the protocol type layer by layer, quickly locks the protocol characteristics of the data stream, avoids performing high-complexity in-depth parsing on all traffic, significantly reduces the consumption of computing resources, and is especially suitable for large-scale traffic analysis in a high-speed network environment. When facing new protocols or complex protocols, the new protocol characteristics or patterns can be dynamically extended to the pattern matching identification and advanced protocol matching stages, without reconstructing the entire identification process. Each stage is independently optimized, and the update and maintenance costs are low, and the ability to adapt to new protocols is strong. Description of the Drawings

[0049] Figure 1 It is a schematic flowchart of the network protocol parsing method provided by the embodiment of the present invention;

[0050] Figure 2 It is a schematic block diagram of the parsing system based on the three-plane separation architecture provided by the embodiment of the present invention;

[0051] Figure 3 It is a schematic flowchart of the feedback-based two-level equalization routing algorithm provided by the embodiment of the present invention;

[0052] Figure 4 It is a schematic flowchart of the feedback-based dynamic hash algorithm provided by the embodiment of the present invention;

[0053] Figure 5 It is a schematic flowchart of the lookup session flow table based on the hash algorithm provided by the embodiment of the present invention;

[0054] Figure 6It is a principle block diagram of a high-speed DPI engine based on a three-stage pipeline architecture provided by an embodiment of the present invention;

[0055] Figure 7 It is a principle block diagram of four metadata extraction models provided by an embodiment of the present invention. Detailed implementation manners

[0056] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0057] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention.

[0058] It should be noted that, without conflict, the features in the embodiments of the present invention can be combined with each other.

[0059] In the existing network traffic protocol identification and parsing system, most of its server architectures adopt a multi-CPU, multi-core two-level SMP architecture and an internal shared architecture. Therefore, for large-bandwidth network traffic, when high-throughput data needs to be processed, when a large number of data streams need to enter or leave the memory simultaneously, it will cause congestion in the memory channels, restricting the fast input and output of data; the resource scheduling overhead increases, affecting the real-time performance and performance of data stream processing; due to multiple CPU cores sharing the memory, there will be memory access conflicts, especially in the case of high concurrency, which leads to a decrease in memory access efficiency and affects the speed and efficiency of data processing; and in the architecture with a shared memory channel, data frame forwarding processing and protocol parsing calculation will compete for memory bandwidth and CPU resources, which will lead to instability and increased latency in data processing.

[0060] Due to the influence of the above factors, the existing multi-CPU, multi-core shared memory channel architecture is difficult to effectively meet the continuous processing requirements above 40 Gbps, especially in the case of high load and high concurrency.

[0061] Furthermore, in order to improve the various defects in the above-mentioned prior art, the embodiments of the present invention propose a method and system for identifying and parsing large-bandwidth network traffic protocols, which can adapt to the situation of rich existing network applications, complex types and structures of network protocols, and fast changes, and achieve accurate data processing and network protocol parsing.

[0062] First of all, the embodiments of the present invention provide a system for identifying and parsing large-bandwidth network traffic protocols. This system is based on a high-speed network signal software processing architecture with separation of data, control, and computing planes. Among them, the data plane is responsible for the actual processing of network data forwarding, management, and protection; the control plane is responsible for managing the execution of control commands, updating the data flow table, and providing rule bases; the computing plane is responsible for executing computing functions such as protocol identification and parsing, and metadata extraction.

[0063] More specifically, communication is achieved between the data plane and the control plane through the execution of control commands and the update of the data flow table. Communication is achieved between the control plane and the computing plane through data transfer. The control plane provides rules and bases, and the computing plane executes corresponding computing tasks.

[0064] Reference Figure 2 , Figure 2 is the principle block diagram of the parsing system based on the three-plane separation architecture provided by the embodiments of the present invention. In the architecture design of three-plane separation, the data plane, the control plane, and the management plane operate independently, with clearly defined functions and responsibilities. The data plane forwards different data to the computing plane through several parallel processing layers via different transmission channels. The control plane is responsible for the management of network policies and the distribution to the data plane and the computing plane. After receiving different types of data forwarded by the data plane, the computing plane executes computing functions such as protocol identification and parsing, and metadata extraction.

[0065] It can be understood that in the existing parsing system with a non-separated three-plane architecture, an internally shared architecture is adopted, and the functions of each plane are more integrated, without a clear logical division. The management and distribution of data may tend to be completed in the same device or system. Therefore, in the distribution and processing of data, there will be a situation where the shared channel is blocked when processing large data, which will affect the accuracy and recognition efficiency of subsequent protocol recognition results. In the three-plane separation architecture of the parsing system provided by the embodiments of the present invention, the data distribution logic is clearer and more hierarchical, and there is a clear data flow path between each plane. Its design concept can better adapt to the growing data requirements and complex network environments. Through the mutual cooperation of the three planes, the normal operation of the network, the effective forwarding of data packets, and the monitoring and management of network devices are realized. This separation architecture can improve the scalability, reliability, and security of the network, while maintaining the independence and function focus of each plane.

[0066] Based on the design architecture of the triple-plane separation of the above parsing system, for how to evenly distribute data in network signal processing, how to identify network protocols, and how to extract metadata, an embodiment of the present invention provides a method for identifying and parsing large-bandwidth network traffic protocols, in combination with reference Figures 1 to 3 , and its working process is as follows:

[0067] Step 101: Receive the original traffic of the network and store it in the buffer in the kernel state in the form of data frames. For example, in some embodiments of the present invention, the network interface card (NIC) directly receives high-speed traffic data from the external network through hardware. These data are encapsulated in the Ethernet frame format and stored in the buffer in the operating system kernel state. To meet the requirements of high bandwidth and low latency, the kernel-state buffer adopts a multi-queue architecture, and the RSS technology is used to achieve the initial dispersion of traffic. Each queue independently processes a part of the traffic, thus avoiding data competition between queues and improving the reception efficiency.

[0068] Step 102: Based on the zero-copy technology, directly map the data frames in the kernel-state buffer to the user-state memory area, which serves as the starting point for the data stream to enter the user-state processing. For example, in some embodiments of the present invention, the DPDK technology is used to establish a direct memory mapping between the user state and the kernel state. The zero-copy technology ensures that when the traffic data is transferred from the kernel-state buffer to the user state, no additional data copying is performed, reducing the memory bandwidth consumption and CPU overhead. This process is also based on the NUMA optimization strategy, which allocates memory resources to the area with the closest physical distance to the computing core, thereby reducing the memory access latency.

[0069] Step 103: According to the principle of the same source and the same destination, logically classify the data stream based on the source address, destination address, source port, and destination port, and send the classified data stream to the corresponding computing processing core through the dynamic distribution mechanism; among them, the dynamic distribution mechanism distributes the data stream to different computing processing cores through the load balancing strategy. For example, in some embodiments of the present invention, in the user-state memory, the system logically classifies the data stream according to the traffic characteristics (such as source IP, destination IP, source port, destination port, protocol type, etc.). The hash algorithm is used to calculate the hash value of the traffic classification to ensure that the data frames of the same source session are assigned to the same computing core (the principle of the same source and the same destination). The distribution process is completed through primary load balancing and secondary load balancing.

[0070] Primary load balancing: Distribute the received traffic to different CPU core channels according to the data volume.

[0071] Secondary load balancing: Utilize the dynamic distribution mechanism to dynamically adjust the traffic distribution weight according to the real-time load status of the computing core, avoiding load imbalance and single-point bottlenecks.

[0072] Step 104: Each computing and processing core sequentially completes layer-by-layer parsing of the allocated data stream through a three-stage pipelining architecture; among them, the three-stage pipelining architecture includes three parsing stages: underlying protocol recognition, pattern matching recognition, and high-level protocol matching recognition. Underlying protocol recognition determines the underlying protocol type by parsing link layer and network layer fields. Pattern matching recognition parses the middle-level protocol through field matching rules and a predefined pattern library. High-level protocol matching recognition parses complex protocols by combining context information and state machine mechanisms. After the data stream completes parsing in each stage, if the protocol type is determined, the result is directly output; otherwise, it enters the next stage for parsing.

[0073] Specifically, in some embodiments of the present invention, in the computing and processing core, the data stream enters the protocol parsing module, and this module sequentially performs the following parsing in the manner of a hierarchical protocol.

[0074] Underlying protocol recognition: Recognize information such as MAC addresses and frame types in Ethernet frames.

[0075] Pattern matching recognition: Parse network layer protocols (such as IPv4, IPv6) and transport layer protocols (such as TCP, UDP), and extract fields such as IP addresses, port numbers, and sequence numbers.

[0076] High-level protocol matching: Deeply parse application layer protocols (such as HTTP, DNS, FTP, etc.), and extract complete protocol fields and data content.

[0077] Step 105: Based on the protocol recognition result, call an adapted metadata extraction model in each computing and processing core to extract metadata from the recognition result; among them, each metadata extraction model is bound to protocol characteristics and is automatically matched and called according to the result of the protocol parsing stage. For example, in some embodiments of the present invention, after the protocol parsing is completed, the system calls an adapted metadata extraction model to extract specific features from the data, such as:

[0078] For the HTTP protocol, extract metadata such as URL, request method, status code, etc.;

[0079] For the DNS protocol, extract metadata such as query domain name, response record, etc.;

[0080] For the FTP protocol, extract metadata such as file transfer status, file name, etc.

[0081] These metadata extraction models are predefined according to protocol characteristics, automatically select the optimal extraction logic for the parsing results of different protocol layers, and the extracted metadata is used for traffic classification, monitoring, and subsequent application analysis.

[0082] Step 106: Each computing and processing core combines the protocol recognition result and the metadata extraction result into a data queue and then outputs it. For example, in some embodiments of the present invention, after completing protocol recognition and metadata extraction, the system integrates the parsing result and the metadata result into a unified data structure to form a data queue. The structure of the data queue includes: protocol parsing result (hierarchical field data), metadata extraction result (specific feature values), and additional information such as timestamp and traffic size.

[0083] Finally, the data queue is sent to subsequent processing modules for storage, further analysis, or forwarding operations. The data output process supports multiple formats, such as JSON or Protobuf, to adapt to different downstream application scenarios.

[0084] In summary, it can be understood that this parsing method completes the reception, distribution, parsing, and metadata extraction of network traffic, realizes efficient large-bandwidth data processing capabilities, and at the same time provides complete data support for subsequent network monitoring, threat detection, and data analysis.

[0085] Regarding the zero-copy technology implemented in step 102, more specifically, in some feasible embodiments of the present invention, combined with Figure 3 , the zero-copy technology is specifically the MFRU header data prefetch management algorithm based on zero-copy. This algorithm adopts a two-level table structure of a first-level free table stack and a second-level doubly linked free list. When the network card puts the received data frame (including the header and payload) into memory through the DPDK technology, the buffer pool mechanism of the MFRU algorithm will specifically manage the header data. Among them, the header data is the key part of data frame parsing and processing (for example, protocol parsing, data stream distribution, etc.). The MFRU algorithm constructs an efficient buffer pool to centrally manage the unprocessed headers, avoiding frequent memory allocation and recycling overhead.

[0086] Regarding the hierarchical management of the two-level table structure, it is as follows.

[0087] The first-level free table stack is used to manage all unallocated table entries, which are memory blocks for storing header data and are organized in pages to ensure the efficiency of batch allocation and recycling. For example, during peak network traffic periods, when more memory needs to be allocated, the first-level stack will allocate table entries by paging.

[0088] The second-level doubly linked free list is used to allocate and recycle header data buffers. It is a structure directly serving data frame processing and stores the currently used or to-be-allocated header buffers. Its doubly linked lock-free queue design avoids lock contention in a multi-threaded environment and improves multi-core parallel processing performance.

[0089] As described above, it can be understood that when distributing data frames, the system needs to determine the data stream characteristics based on fields such as the source IP, destination IP, and port number in the packet header. At this time, the MFRU algorithm ensures that this packet header data is quickly extracted and stored in the buffer pool for efficient access by the distribution logic, and the two-way lock-free queue structure further reduces the latency during access, ensuring the real-time nature of the data stream distribution logic.

[0090] In addition, during the distribution process, the feedback mechanism needs to dynamically adjust the data stream allocation strategy, such as adjusting according to the load or queue length. The MFRU algorithm enables the distribution module to efficiently manage and schedule memory resources by quickly allocating and recycling packet header buffers, reducing the impact of memory management on the performance of the routing algorithm.

[0091] It can be understood that the core goal of zero-copy is to avoid redundant memory copy operations. Once the data frame is received and stored in memory, all subsequent operations (such as protocol parsing, distribution, and processing) are completed through pointer operations. The MFRU packet header data prefetch management algorithm based on zero-copy ensures efficient and stable allocation and recycling of packet header data through a two-level table structure, providing memory management support for zero-copy. The packet header buffer pool is responsible for managing the packet header data stored in memory, ensuring that this data can be transparently accessed in user-mode processes. The lock-free design avoids competition between the kernel mode and the user mode, greatly reducing context switching and memory access latency.

[0092] Regarding the allocation process of empty table entries and dynamic memory management, in some feasible embodiments of the present invention, the MFRU packet header data prefetch management algorithm based on zero-copy adopts a two-level table structure, constructing a packet header data structure buffer pool equal to the number of buffer pools. Each packet header data structure is a table entry, with a size of 32 bytes and aligned according to the 128-bit boundary, so as to ensure that each packet header data structure is exactly a multiple of the bit widths of the cache line and the memory channel bank, maximizing the utilization of the cache and memory characteristics. By storing this information in the packet header data structure and organizing it in a specific alignment manner, the access efficiency of this information can be improved, and it is also beneficial to the utilization of the cache.

[0093] More specifically, the MFRU algorithm uses a two-level packet header information free list management strategy. The first level is a large-capacity free list stack that stores all free list entries to be allocated to the second level. It adopts a paged design. In this design, the free list stack is divided into multiple pages, and each page contains a certain number of entries. Here, the size of each page is 64K entries, which means the free list stack is divided into multiple pages of size 64K. This paged design helps organize and manage a large number of free list entries to improve the efficiency of allocation and recycling. Adopting a stack algorithm means that each time a free list entry is allocated, the last (newest) allocated page will be selected to ensure that the most frequently used page is allocated each time. This can improve the cache hit rate because frequently used pages are likely to be used again, and these pages can be accessed more quickly through the stack algorithm.

[0094] The second level is a doubly linked free list, which is directly used to allocate and recycle the buffer queues used by unprocessed processes. It is constructed according to the number of protocol processing processes and adopts a doubly linked lock-free queue structure. The queue size is between a first set threshold and a second set threshold. In some feasible embodiments of the present invention, the queue size is between 32K and 128K entries. 32K as a threshold means that the system can effectively manage and allocate resources within this range of free list entries while maintaining a high cache hit rate; when it exceeds 128K, free list entries are released to avoid excessive occupation of memory resources while maintaining the stability and performance of the system.

[0095] Based on the above, regarding the handling when the number of empty entries is less than 32K and greater than 128K:

[0096] When the number of empty entries is lower than the 32K threshold, the system will page in from the first-level page table to obtain more free list entries, and then insert these free list entries into the second-level doubly linked free list.

[0097] When the number of empty entries is greater than 128K, the system will release a 64K-sized free list page and push it onto the first-level stack to release space and maintain a reasonable memory utilization rate.

[0098] It can be understood that through this strategy of dynamically managing free list entries, the system can better handle data frame exchanges under different loads and improve data exchange efficiency and performance.

[0099] Now take the Figure 3 system as an example. Assume that the system is processing a high-load network traffic scenario:

[0100] The network card receives data frames through DPDK. The packet headers and payloads in the frames are stored in memory. The MFRU algorithm allocates an idle entry in the buffer pool to store the packet header data. The system extracts the packet header data from the buffer pool, parses information such as its source IP, destination IP, port number, etc. According to the parsing results, the system distributes the data stream to the specified processing core according to the same-source and same-destination rule.

[0101] If the number of idle entries in the buffer pool is insufficient (below the first-level threshold), the algorithm pages in from the first-level stack to expand the number of available entries; if the decrease in traffic causes too many idle entries (exceeding the second-level threshold), the algorithm reclaims the excess entries into the first-level stack. After the data frame processing is completed, the entry of the packet header data is reclaimed into the buffer pool for the next batch of data frames to use.

[0102] Regarding how the classified data stream is sent to the corresponding computing processing core through the dynamic distribution mechanism in step 103, more specifically, in some feasible embodiments of the present invention, a feedback two-level equalization routing algorithm is adopted for the two-level distribution of the data stream. This algorithm adopts a two-level routing equalization distribution method, and sends the network data stream to be processed to the computing processing core for parallel processing according to the principle of the same source and the same destination. The computing processing core feeds back the traffic monitoring data to the second-level equalization distribution core, and the second-level equalization distribution core adjusts the hash algorithm in real time according to the detection data to achieve dynamic equalization of the traffic.

[0103] On the above basis, refer to Figure 3 , Figure 3 is a schematic diagram of the working process of the feedback two-level equalization routing algorithm provided by the embodiment of the present invention; the first level adopts a multi-interface load cross-equalization routing algorithm to achieve the balanced sending of the input data of multiple network cards to 4 CPU channels, avoiding congestion packet loss caused by unbalanced traffic between network cards, and at the same time making full use of the characteristics of the NUMA architecture to improve the processing performance. The second level adopts a feedback dynamic equalization hash algorithm to solve the problem of evenly distributing the data stream in the CPU channel to each processing core.

[0104] Specifically, network data streams are received through multiple NICs. Each NIC distributes the received data streams into multiple Receive Side Scaling (RSS) queues for preliminary load balancing. Through the Direct Memory Access (DMA) technology, the data is transferred from the RSS queues of the NICs to the corresponding Non-Uniform Memory Access (NUMA) memory regions. The NUMA memories are NUMA Memory 1, NUMA Memory 2, NUMA Memory 3, and NUMA Memory 4 respectively, and each NUMA memory region corresponds to a different CPU. The data in each NUMA memory region is transferred to the corresponding distribution queue for further processing. In each CPU, including the second-level balanced distribution core and the computing processing core, the load conditions of the computing processing cores during protocol parsing and metadata extraction are monitored, and the monitoring data is fed back to the second-level balanced distribution core. The monitoring data includes information such as the current load and processing efficiency of each computing processing core.

[0105] Based on the monitoring data, the second-level balanced distribution core dynamically adjusts the hash algorithm and distributes the data streams to the appropriate computing plane processing cores. Regarding how to dynamically adjust the hash function, refer to Figure 4 , Figure 4 which is the schematic diagram of the operation process of the feedback-based dynamic hash algorithm provided by the embodiment of the present invention, specifically including the following judgment steps.

[0106] Judge whether the data streams received by the processing cores are continuously unbalanced and overloaded. Yes: Enter S201: The processing core real-time statistics the current session number and growth rate and feeds them back to the distribution processing core; No: End.

[0107] Judge whether the current session number and growth rate exceed the threshold. Yes: Enter S202: The distribution core reselects the hash function; No: End.

[0108] Judge whether the same-source and same-destination of the data streams are guaranteed after the hash function is switched. Yes: Enter S203: When the data packet arrives, calculate two hash values and retrieve the data stream table on the meta-hash routing path; No: End.

[0109] Judge whether the old hash algorithm value is hit in the data stream table. Yes: Enter S204: Route according to the original hash algorithm value; No: End.

[0110] Judge whether the timeout time has been exceeded and there is no new addition after the flow table update is completed. Yes: Enter S206: Perform routing calculation according to the new hash algorithm; No: End.

[0111] Judge whether there is no hit of the old route within the set time window. Yes: It means that the hash function switch is completed; No: End.

[0112] The hash algorithm is dynamically adjusted. After the data stream is distributed to each computing and processing core, each computing and processing core processes the allocated data stream in parallel. The processed data is sent back to the network through the output queue and NIC.

[0113] In summary, through the two-level balanced distribution method, the feedback-based two-level balanced routing algorithm realizes the efficient parallel processing of network data streams. By dynamically adjusting the hash algorithm, it ensures the balanced distribution of traffic and the efficient operation of the system.

[0114] Protocol parsing and metadata extraction are carried out at the granularity of session streams. Therefore, the access to the flow table is one of the operations with the highest frequency and is also the key optimization bottleneck operation. In the embodiments of the present invention, the flow table management layer adopts a distributed hash flow table management algorithm based on the Splay tree. This algorithm is different from traditional sharing technologies (such as shared memory or shared database). By distributing the flow table on each processing core through the distributed hash flow table management algorithm based on the Splay tree, independence is achieved, and competition for shared resources is avoided. The flow table is distributed on each processing core, and the scale of a single flow table is greatly reduced, making the management burden on each processing core lighter and improving the processing efficiency. Since the flow tables are independent of each other and do not share, this algorithm can better utilize the parallelism of the processing cores, thereby improving the overall processing capacity and throughput.

[0115] Regarding how to search the session flow table, in some feasible embodiments provided by the present invention, refer to Figure 5 , and the specific process is as follows.

[0116] Step 301: Select a hash function and a loading factor; that is, select according to specific scenarios and data characteristics.

[0117] Step 302: Calculate the hash value of the flow table entry to be searched through the hash function, and search for the corresponding flow table entry in the hash table according to the hash value. The system first searches for the corresponding position in the hash table according to the calculated hash value. If the flow table entry is found at this position, the result is directly returned. If a hash conflict occurs, go to Step 303.

[0118] Step 303: When a hash conflict occurs at the corresponding position in the hash table, store the data with the hash conflict in the Splay tree structure queue, and repeatedly compare the hash value of the flow table entry to be searched with the nodes of the Splay tree until the target flow table entry is found. That is, in the hash table, multiple data are mapped to the same position, and the system needs to further process. At this time, the system can search for the conflicting data through the Splay tree structure queue.

[0119] The system will search for the stored conflicting data in the Splay tree structure queue. A Splay tree is a self-adjusting binary search tree. Through rotation operations, the frequently accessed nodes can be adjusted to the root node position, thereby improving the search efficiency.

[0120] Once the flow table entry corresponding to the conflicting data is found in the Splay tree, the system can return the corresponding result.

[0121] Among them, regarding how to search in the Splay tree: Before performing the search, a series of Splay operations need to be executed first to move the data item to be searched to the root node of the tree. Once the target node (i.e., the hash value of the flow table entry to be searched) is moved to the root node, the system will compare the hash value of the flow table entry to be searched with the value of the current root node. According to the comparison result, the system will move the target node to the appropriate position along the node movement path of the Splay tree. If the value of the target node is less than the value of the current node, it will move to the left subtree; if it is greater than the value of the current node, it will move to the right subtree. The system will repeat the comparison and node movement operations until the target node is found or it is determined that the target node does not exist in the tree. Once the target node is found, the system will perform a final position adjustment to ensure that the target node is located at the root node position of the tree. In this way, the target node can be accessed more quickly when searching for the same data next time.

[0122] On the above basis, considering that the frequent deletion operations of session flows will also cause bottleneck problems in flow table access, therefore, in some feasible embodiments of the present invention, a doubly linked list is used to manage the queue of online flow tables. According to the locality principle, new session flows are enqueued using the tail insertion method. The timeout scan is not performed in the form of a timer, but is driven by the number of arriving data packets. Each time, a part of the table is scanned to smooth the computing resources occupied by the timeout scan.

[0123] The following is an embodiment of optimizing the use of a doubly linked list queue management using Python:

[0124] def _init_(self, data):

[0125] self.data = data

[0126] self.prev = None

[0127] self.next = None

[0128] class DoublyLinkedList:

[0129] def_init_(self):

[0130] self.head = None

[0131] self.tail = None

[0132] def append(self, data):

[0133] new_node = Node(data)

[0134] if self.head is None:

[0135] self.head = new_node

[0136] self.tail = new_node

[0137] else:

[0138] new_node.prev = self.tail

[0139] self.tail.next = new_node

[0140] self.tail = new_node

[0141] def remove(self, node):

[0142] if node.prev:

[0143] node.prev.next = node.next

[0144] else:

[0145] self.head = node.next

[0146] if node.next:

[0147] node.next.prev = node.prev

[0148] else:

[0149] self.tail = node.prev

[0150] doubly_linked_list = DoublyLinkedList()

[0151] def packet_arrival_handler(packet):

[0152] # Logic for processing data packets

[0153] if packet.arrival_condition:

[0154] perform_timeout_scan()

[0155] def perform_timeout_scan():

[0156] # Logic for performing timeout scan, processing partial flow table entries each time

[0157] In the face of the problems that the existing types and structures of network protocols are complex, change rapidly, it becomes more difficult to ensure the accuracy and completeness of protocol type parsing, and it is difficult to perform real-time parsing and metadata extraction under high load and high concurrency, in the embodiments provided by the present invention, the computing plane uses five protocol recognition methods based on different parameters and features. Regarding the five recognition methods based on different parameters and features, they are as follows.

[0158] Underlying protocol recognition method, responsible for performing preliminary recognition based on the basic features of data packets and protocol header information; by using underlying protocol analysis techniques, such as port recognition, protocol identifiers, etc., output including the preliminarily recognized protocol type or features.

[0159] Pattern matching recognition method, by comparing the content of data packets with predefined patterns for matching to confirm the protocol type; by using pattern matching algorithms, such as regular expression matching, signature matching, etc., output the confirmed protocol type or features.

[0160] Logically related data packet recognition method, by analyzing the logical association relationships between data packets to further determine the protocol type; by using logical association analysis techniques, such as TCP / IP connection state analysis, data packet sequence relationships, etc., output the further confirmed protocol type or features.

[0161] Multi-association layer matching method, combining the information and association relationships of multiple data packets to comprehensively judge the protocol type of data packets; by using multi-association layer matching algorithms, such as protocol interaction pattern matching, multi-traffic feature analysis, etc., output the comprehensive recognition result of data packets.

[0162] Statistical analysis feature method, finally, the data packet enters the statistical analysis feature model for further feature analysis and confirmation. This model finally determines the protocol type of the data packet through statistical analysis of the data packet features and outputs the finally confirmed protocol type or features.

[0163] Based on the above five protocol recognition methods, considering the complexity of the recognition algorithm, a hierarchical iterative calculation is implemented, with the host progressing precisely. In some feasible embodiments of the present invention, a high-speed DPI engine based on a three-stage pipelined matching architecture is designed and implemented. The data stream passes through the three-stage pipelined architecture in sequence, that is, through the underlying protocol recognition, pattern matching recognition, and advanced protocol matching, to complete the recognition and parsing of the protocol at one time. Refer to Figure 6 , Figure 6 FIG. is the principle block diagram of the high-speed DPI engine based on the three-stage pipelined architecture provided by the embodiment of the present invention, which is specifically as follows.

[0164] The underlying protocol recognition method is used as the first-level recognition to recognize the underlying protocols with few and accurate rules. By extracting the IP header information and port numbers and performing algorithm recognition based on the 3 / 4 layer features, fast and accurate protocol recognition is achieved.

[0165] The pattern matching recognition method is used as the second-level recognition to handle more complex protocol recognition requirements. Through the pattern matching algorithm, the patterns in specific data packets are recognized, thereby identifying the corresponding protocol types and achieving more refined protocol recognition and processing.

[0166] The logical association data packet recognition method, multi-association layer matching method, and statistical analysis feature method are used as the third-level recognition and are processed using regular matching, logical model matching, and statistical analysis recognition algorithms. By analyzing multi-level associations, logical associations, and statistical features, complex protocol types are recognized. These algorithms combined can improve the accuracy and efficiency of protocol recognition.

[0167] Based on the above, it can be understood that the data packet first enters the underlying protocol recognition algorithm (first-level recognition) for fast and accurate rule recognition. After the first-level recognition is completed, the recognition result is passed to the pattern matching recognition algorithm (second-level recognition) for more complex protocol recognition and processing. Then the data packet is passed to the third-level recognition, including the logical association data packet recognition algorithm, multi-association layer matching model, and statistical analysis feature model, to handle more complex protocol recognition requirements.

[0168] Through the three-stage pipelined matching architecture, the data stream can pass through the pipelined architecture at one time, and through the three-level recognition process of hierarchical iterative calculation, the accurate recognition and parsing of the data packet are achieved, thereby greatly improving the efficiency and accuracy of protocol recognition. This design can effectively cope with different levels of complex protocol recognition requirements and provide a more efficient and accurate solution for network data processing.

[0169] In order to achieve the fast extraction of different types of metadata, according to the extraction method and hierarchical granularity, in the embodiments of the present invention, four disjunctive method models are constructed in the computing plane, specifically refer to Figure 7 , Figure 7It is the principle block diagram of four metadata extraction models provided by the embodiments of the present invention.

[0170] The packet-driven metadata extraction model extracts the IP header and data part in the data packet, analyzes the data part and then extracts relevant metadata for output. For example, the extraction of OSPF protocol packets.

[0171] The flow-driven metadata extraction model extracts data packets from the data stream, then extracts the IP header and data part of each data packet, analyzes the entire flow data and then extracts relevant metadata for output. For example, the extraction of the username and password in the Telnet protocol.

[0172] The tunnel logical session-driven metadata extraction model extracts logical session information from the tunnel protocol data, then extracts data packets, extracts the IP header and data part from the data packets, analyzes the logical session data and extracts relevant metadata for output. For example, the extraction of the session ID of the L2TP protocol.

[0173] The event-driven metadata extraction model extracts the IP header and data part in the data, then analyzes according to the protocol requirements, and extracts corresponding metadata according to the analysis results for output. For example, the metadata extraction in the ESP protocol.

[0174] As can be seen from the above architecture, the pattern matching algorithm is the most frequently used algorithm both in the protocol recognition process and in the metadata extraction process, and its efficiency determines the real-time recognition performance of various applications. In order to implement multi-pattern line speed matching processing of various applications with a time complexity of O(n), in some feasible embodiments of the present invention, a multi-pattern matching algorithm based on perfect hashing is designed at the second-level recognition.

[0175] Specifically, since the protocol feature rule library used for matching is static, the library can be preprocessed at compile time to construct a minimum perfect hash function for it. If a hash function satisfies that for any different key values and , if and only if when, there is , it is a perfect hash function, that is, when hashing a key value set L, no conflicts are generated.

[0176] Suppose there is a static protocol feature rule library, which contains the following rules, and these rules are used as the key value set L:

[0177] Rule 1: Protocol type: HTTP

[0178] Rule 2: Protocol type: FTP

[0179] Rule 3: Protocol type: SSH [[ID=3,9]]

[0180] Construct a key-value set:

[0181]

[0182] Select appropriate parameters and to construct a hash function, usually a non-zero integer randomly selected;

[0183] For a given key-value , calculate the hash value through the hash function :

[0184]

[0185] where is the size of the hash table, usually determined according to the number of patterns to be stored; is a large prime number, usually taken large enough to avoid collisions;

[0186] For Rule 1: Protocol type: HTTP

[0187]

[0188] For Rule 2: Protocol type: FTP

[0189]

[0190] For Rule 3: Protocol type: SSH

[0191]

[0192] Here, use the ASCII value of each character to replace the key-value. After calculating the hash value, store these rules in the hash table to ensure that each hash value corresponds to a unique rule.

[0193] Next, store these hash values in the hash table for fast pattern matching during the data stream matching process.

[0194] During the data stream matching process, perform hash calculation on parts of the data stream and look up whether there is a corresponding hash value in the hash table to achieve pattern matching.

[0195] On the above basis, regarding the matching of the rule library, in some feasible embodiments of the present invention, the AC algorithm is adopted. The AC algorithm is a multi-pattern matching algorithm used to find the occurrence positions of multiple pattern strings in the text, and the minimum perfect hash function is used for fast matching and identifying specific rules or keywords.

[0196] First, use the AC algorithm to construct an automaton that contains all the pattern strings to be matched (strings in the rule base). The construction process of the AC automaton can ensure efficient searching for the occurrence positions of multiple pattern strings in the text.

[0197] Design the minimal perfect hash function with reference to the above steps.

[0198] For each string to be matched, first use the AC automaton to perform matching in the text to find the matching positions.

[0199] Then use the minimal perfect hash function to verify the matching positions to ensure that the matched string is a valid string in the rule base.

[0200] In some feasible embodiments of the present invention, regarding the generation of the matching string, instead of performing a full match on the data packet, 12 dimensions of the first 5 frames of each protocol are selected for splicing 256 bytes of data. Among them, the selection of dimensions is based on the partial position data of the most frequently occurring feature words in the protocol. Use this string for MPHF-AC algorithm processing to obtain a matching vector of multiple feature words. This vector includes position information and the string label of the successful match. Such a design realizes the rapid identification of the features of a specific protocol in the data packet without the need to perform a full match on the entire data packet. By selecting key dimensions and feature words, the protocol features can be effectively identified and matched, and the MPHF-AC algorithm can efficiently process these matches to obtain position information and the string label of the successful match.

[0201] In addition, considering that if the values of the matching vector conform to multiple protocol feature items, in some embodiments of the present invention, the DFA algorithm is used to further verify and identify the specific protocol. The DFA algorithm can accurately identify whether the input conforms to a specific pattern or rule according to the predefined state transition rules and termination states.

[0202] The above is only the preferred embodiment of the present invention and is not intended to limit the present invention. The scope of patent protection of the present invention is subject to the claims. All equivalent structural changes made by using the content of the specification and drawings of the present invention should, by the same token, be included within the protection scope of the present invention.

Claims

1. A method for identifying and parsing large-bandwidth network traffic protocols, characterized in that, It includes the following steps: Receive the original traffic of the network and store it in the buffer in the kernel state in the form of data frames; Based on the zero-copy technology, directly map the data frames in the kernel-state buffer to the user-state memory area as the starting point for the data stream to enter the user state for processing; According to the principle of the same source and the same destination, logically classify the data stream based on the source address, destination address, source port, and destination port as the classification basis, and send the classified data stream to the corresponding computing processing core through the dynamic distribution mechanism; among them, the dynamic distribution mechanism distributes the data stream to different computing processing cores through the load balancing strategy; Each of the computing processing cores sequentially completes layer-by-layer parsing of the allocated data stream through a three-stage pipeline architecture; among them, the three-stage pipeline architecture includes three parsing stages: underlying protocol recognition, pattern matching recognition, and advanced protocol matching recognition. The underlying protocol recognition determines the underlying protocol type by parsing the link layer and network layer fields. The pattern matching recognition parses the middle-layer protocol through the field matching rule and the predefined pattern library. The advanced protocol matching recognition parses the complex protocol by combining the context information and the state machine mechanism; after the data stream completes parsing in each stage, if the protocol type is determined, the result is directly output, otherwise it enters the next stage for parsing; Based on the recognition result of the protocol, call the adapted metadata extraction model in each of the computing processing cores to extract metadata from the recognition result; among them, each of the metadata extraction models is bound to the protocol characteristics and is automatically matched and called according to the result of the protocol parsing stage; Each of the computing processing cores combines the protocol recognition result and the metadata extraction result into a data queue and then outputs it.

2. The method for identifying and parsing a large bandwidth network traffic protocol according to claim 1, wherein The zero-copy technology is specifically the MFRU header data prefetch management algorithm based on zero-copy. This algorithm adopts a two-level table structure of the first-level free list stack and the second-level double-linked free list, and allocates a free list entry in the kernel-state buffer to store the header data in the data frame; specifically: When the number of free list entries in the buffer is lower than the set first set threshold, this algorithm invents page faults to the first-level free list stack to expand the number of available list entries; When the number of free list entries in the buffer is greater than the set second set threshold, this algorithm releases the free list entries and reclaims them into the first-level free list stack.

3. A method for identifying and parsing a large-bandwidth network traffic protocol according to claim 1, characterized in that, The load balancing strategy is specifically to adopt a feedback-based two-level equalization routing algorithm for two-level distribution of the data stream. Specifically: The first level sends the data stream to the computing nodes according to the principle of the same source and the same destination by obtaining the real-time load information of each computing node according to the load situation; The second level dynamically adjusts the hash algorithm inside each computing node based on the load monitoring data of the computing processing cores and then distributes the data stream to each computing processing core again to achieve balanced distribution of the data stream within the computing node.

4. A method for identifying and parsing a large-bandwidth network traffic protocol according to claim 1, characterized in that, During the identification and parsing process of the protocol, a distributed hash flow table management algorithm based on the Splay tree is used to construct a distributed flow table according to the number of the computing processing cores. Each computing processing core independently maintains a flow table in the form of a Splay tree structure. When the data stream arrives at the computing processing core, the corresponding flow table entry will be searched in the Splay tree; If the corresponding flow table entry is found, the protocol status stored in the flow table will be directly obtained; If the corresponding flow table entry is not found, a new flow table entry will be created and inserted into the Splay tree.

5. The method for identifying and parsing a large bandwidth network traffic protocol according to claim 1, characterized in that The underlying protocol identification and parsing stage in the three-level pipeline architecture is responsible for quickly identifying the link layer and network layer protocol types of the data packet, which serves as the basic stage of protocol parsing. Specifically, in this stage, preliminary identification is carried out based on the basic characteristics of the data packet and the protocol header information to identify the data link layer and the network layer; If the link layer or network layer protocol type is successfully identified, the protocol identification result will be directly output and the data stream parsing will be completed; If the protocol type cannot be determined, the data stream will enter the pattern matching identification and parsing stage of the next stage.

6. A method for identifying and parsing a large-bandwidth network traffic protocol according to claim 5, characterized in that The pattern matching identification and parsing stage in the three-level pipeline architecture is responsible for further identifying the transport layer protocol type of the data packet. By comparing the data packet content with the predefined matching patterns, the transport layer protocol type is confirmed. Specifically, a minimum perfect hash function is constructed to map the predefined matching patterns to unique hash values, avoiding hash conflicts and achieving fast matching. Regular expression matching and signature matching are used to locate the protocol feature fields; If the transport layer protocol type is successfully confirmed, the protocol identification result will be directly output and the data stream parsing will be completed; If the protocol type cannot be determined, the data stream will enter the advanced protocol matching of the next stage.

7. A method for identifying and parsing a large-bandwidth network traffic protocol according to claim 6, characterized in that The advanced protocol matching and identification parsing stage in the three-level pipeline architecture is responsible for identifying complex application layer protocol types. This stage combines the logical association data packet identification method, the multi-association layer matching method, and the statistical analysis feature method to identify the application layer protocol; correspondingly: The logical association data packet identification method defines the protocol context by analyzing the logical association relationship between data packets; The multi-association layer matching method judges the application layer protocol type by synthesizing the information and association relationship of multiple data packets; The statistical analysis feature method analyzes the protocol by using traffic statistical features; Output the complete protocol parsing result to complete the parsing of the data stream.

8. A method for identifying and parsing a large-bandwidth network traffic protocol according to claim 1, characterized in that, Based on the protocol identification result, an adapted metadata extraction model is called in each of the computing processing cores. Correspondingly: The packet-driven metadata extraction model extracts relevant metadata after extracting the IP header and data part in a single data packet; The flow-driven metadata extraction model extracts relevant metadata after analyzing the data packets in the entire data stream; The tunnel logic session-driven metadata extraction model extracts logical session information from the tunnel protocol data and extracts relevant metadata after analyzing the logical session data; The event-driven metadata extraction model first extracts the IP header and data part in the data, and extracts the corresponding metadata after analysis according to the protocol requirements.

9. The method for identifying and parsing a large bandwidth network traffic protocol according to claim 2, characterized in that, Each of the computing and processing cores integrates the protocol recognition result and the metadata extraction result into a data queue, and hands over the data queue to a double-ended lock-free linked list for management. Specifically: When the output of the data queue is completed and marked as processed, the computing and processing core stores the processed data queue in the double-ended lock-free linked list; The header table entries in the data queue are recycled through the MFRU header data prefetch management algorithm and re-added to the first-level free list stack for subsequent data streams to use.

10. A parsing system for a large bandwidth network traffic protocol recognition method, which is used to implement the protocol recognition and parsing method described in any one of claims 1 to 9, characterized in that, The parsing system adopts an architecture based on the separation of data, control, and computing planes, where: The data plane is used to receive network traffic data, perform preliminary classification and distribution of the data to ensure that the traffic can enter the user-mode processing; among them, the reception, classification, and distribution of traffic data are completed through the MFRU header data prefetch management algorithm and the second-level feedback-based equalization routing algorithm; The computing plane is used to perform protocol recognition and parsing and metadata extraction on the data stream; among them, the traffic protocol recognition and parsing and metadata extraction are completed through the Splay tree flow table management algorithm, and the data queue is managed in combination with the MFRU header data prefetch management algorithm; The control plane is used to manage resource allocation, policy adjustment, and global load balancing; among them, the flow table optimization and global load balancing are performed through the Splay tree flow table management algorithm and the second-level feedback-based equalization routing algorithm.

Images