System for implementing traffic collection aggregation and load distribution processing

Abstract

The present invention relates to a system for realizing traffic collection aggregation and load distribution processing. Among them, the system is provided with: a distribution cache table for recording the correspondence between IP and tunnel ID in the system cache, and between MAC address and tunnel ID; a traffic collection parsing and distribution module for performing traffic collection, parsing and distribution processing by establishing a distribution tunnel; a distribution cache table update module for, after receiving the IP address or MAC address input by the traffic collection parsing and distribution module, storing the updated distribution cache table into the system memory, sharing it for use by other modules, and forwarding the collected data packets into the corresponding tunnel session based on the updated distribution cache table. By adopting the system for realizing traffic collection aggregation and load distribution processing of the present invention, the accuracy of analysis will not be affected by traffic distribution, making the analysis process no longer a bottleneck, maximizing the utilization of computing resources, and realizing cost reduction and efficiency increase for users.

Description

Technical Field

[0001] The present invention relates to the field of network communication technologies, and particularly to the field of network traffic collection technologies. Specifically, it refers to a system for realizing traffic collection aggregation and load distribution processing. Background Art

[0002] The network serves as the foundation for carrying services, and network traffic serves as the carrier for service interaction. All interactions of service data are completed through network traffic. Through traffic analysis, attack detection, performance analysis, audit analysis, etc. can be achieved. Traffic analysis has become an important means for ensuring service security and stability.

[0003] Existing traffic analysis methods adopt a design where collection and analysis are carried out synchronously. The traffic analysis system directly collects mirror traffic and conducts analysis. Since analysis requires various rule matching and modeling analysis of traffic data packets, when the traffic is relatively large, the process of rule matching and modeling analysis is very time-consuming; especially when multiple different types of analysis need to be performed on the collected traffic, the problem of time consumption is particularly obvious, and this process becomes a bottleneck restricting the analysis performance of a single computing resource.

[0004] For this reason, in the prior art, when traffic mirroring is performed, the traffic can only be mirrored into multiple computing resources, and multiple computing resources analyze it simultaneously. Although this can solve the problem of large traffic analysis, it requires additional costs to achieve.

[0005] In summary, currently, a technology is needed that can achieve unified collection of traffic within the same computing resource and perform load distribution on the collected traffic, so that the analysis module can be horizontally expanded, thereby breaking through the performance bottleneck of analysis and achieving the purpose of doubling the analysis efficiency under the same computing resources, and realizing the function of cost reduction and efficiency improvement. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a system for realizing traffic collection aggregation and load distribution processing.

[0007] To achieve the above purpose, the system for realizing traffic collection aggregation and load distribution processing of the present invention is as follows:

[0008] The system for realizing traffic collection aggregation and load distribution processing is mainly characterized in that the system includes the following:

[0009] A distribution cache table for recording the correspondence between the IP address and tunnel ID in the system cache, and the MAC address and tunnel ID.

[0010] The traffic collection, parsing and distribution module is connected to the distribution cache table, and is used for performing traffic collection, parsing and distribution processing by establishing a distribution tunnel; and

[0011] The distribution cache table update module is connected to the traffic collection, parsing and distribution module, and is used for, after receiving the IP address or MAC address input by the traffic collection, parsing and distribution module, storing the updated distribution cache table into the system memory, sharing it for use by other modules, and forwarding the collected data packets into the corresponding tunnel session based on the updated distribution cache table for use.

[0012] Preferably, the tunnel ID is specifically:

[0013] After the establishment of the tunnel session is completed, a tunnel ID will be assigned to each tunnel.

[0014] Preferably, the traffic collection, parsing and distribution module establishes a distribution tunnel in the following manner:

[0015] By establishing a listening port to listen for requests to establish a tunnel from other processes or containers. Other processes or containers can complete the establishment of the tunnel by connecting to this port and providing a packet identifier to this port, and an appropriate tunnel ID will be assigned.

[0016] Preferably, the traffic collection, parsing and distribution module also synchronizes tunnel information in the following manner:

[0017] Summarize all the established tunnels, group the tunnels using the packet identifier of the tunnel, and sort the tunnels within the same group according to the tunnel ID.

[0018] Preferably, the traffic collection, parsing and distribution module also shares tunnel grouping information in the following manner:

[0019] After completing the synchronization of tunnel information and obtaining tunnel grouping data, store the tunnel grouping data into the system cache and share the cache for use by other modules to achieve the sharing process of tunnel grouping information.

[0020] Preferably, the traffic collection, parsing and distribution module collects mirror traffic through DPDK and collects in-cloud traffic through cloud probe software.

[0021] Preferably, the traffic collection, parsing and distribution module performs traffic parsing processing in the following manner:

[0022] After completing the traffic collection processing, if the obtained data packet is a three-layer or higher protocol data packet, then parse the source IP and destination IP of the data packet; if the obtained data packet is a two-layer protocol data packet, then parse the source MAC and destination MAC of the data packet.

[0023] Preferably, the traffic collection, parsing and distribution module performs traffic distribution processing in the following manner:

[0024] When the parsed data packet is the source IP and the destination IP, compare the two in terms of size, and compare the smaller-valued IP with the distribution cache table to determine whether the IP is in the distribution cache table. If so, directly distribute the traffic to the corresponding tunnel according to the distribution cache table; otherwise, send the IP to the distribution cache table update module to update the distribution cache table, and forward the data packet to the tunnel session corresponding to the tunnel ID in the updated distribution cache table according to the updated distribution cache table to achieve traffic distribution processing.

[0025] Preferably, the traffic collection, parsing and distribution module performs traffic distribution processing in the following manner:

[0026] When the parsed data packet is the source MAC and the destination MAC, compare the two in terms of size, and compare the smaller-valued MAC with the distribution cache table to determine whether the MAC is in the distribution cache table. If so, directly forward the data packet to the tunnel session corresponding to the tunnel ID according to the distribution cache table; otherwise, send the MAC to the distribution cache table update module to update the distribution cache table, and forward the data packet to the tunnel session corresponding to the tunnel ID in the updated distribution cache table according to the updated distribution cache table to achieve traffic distribution processing.

[0027] Preferably, the distribution cache table update module specifically performs the following processing:

[0028] After receiving the IP or MAC input by the traffic collection, parsing and distribution module, first read the tunnel grouping data from the system cache, query the distribution cache table, correspond the tunnels in the distribution cache table with the tunnel data in the tunnel grouping data to obtain the usage times of each tunnel ID under each group, take the tunnel ID with the smallest usage times in each group, and write the corresponding tunnel ID of the IP or MAC into the distribution cache table. If there are multiple tunnel IDs with the smallest usage times in the group, take the one with the smallest ID value, and write the IP or MAC and the obtained tunnel ID into the distribution cache table together.

[0029] By adopting the system for realizing traffic collection aggregation and load distribution processing of the present invention, it is possible to enable various types of analysis modules to perform analysis simultaneously within the same computing resource, and multiple copies of each type of analysis module can be started to jointly complete the work, so as to achieve the goal of doubling or several times the analysis efficiency within the same computing resource; and traffic data packets with the same IP or the same MAC will be distributed into the same analysis tunnel, which can ensure that the analysis module connected to the analysis tunnel obtains the complete traffic of this IP or this MAC, and will not affect the accuracy of the analysis due to the distribution of traffic, making the analysis process no longer a bottleneck, so as to make the best use of the computing resource and achieve cost reduction and efficiency increase for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the process of establishing a distribution tunnel and synchronizing tunnel information of the present invention.

[0031] Figure 2 It is a schematic diagram of the data processing process of the forwarding cache table update module of the present invention.

[0032] Figure 3 It is a schematic diagram of the relationship between the various modules of the system for realizing traffic collection aggregation and load distribution processing of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] In order to more clearly describe the technical content of the present invention, the following will be further described in conjunction with specific embodiments.

[0034] Before detailing the embodiments according to the present invention, it should be noted that hereinafter, the terms "comprising", "including" or any other variant are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0035] Please refer to Figure 3 As shown, the system for realizing traffic collection aggregation and load distribution processing, wherein the system is provided with:

[0036] A distribution cache table for recording the correspondence between the IP address and the tunnel ID in the system cache, and the MAC address and the tunnel ID;

[0037] A traffic collection parsing and distribution module, connected to the distribution cache table, for establishing a distribution tunnel, performing traffic collection, parsing and distribution processing; and

[0038] The distribution cache table update module is connected to the traffic collection, parsing and distribution module, and is used to store the updated distribution cache table in the system memory after receiving the IP address or MAC address input by the traffic collection, parsing and distribution module, share it for other modules to use, and forward the collected data packets into the corresponding tunnel session based on the updated distribution cache table for use.

[0039] As a preferred embodiment of the present invention, the tunnel ID is specifically:

[0040] After the establishment of the tunnel session is completed, a tunnel ID will be assigned to each tunnel.

[0041] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module establishes a distribution tunnel in the following manner:

[0042] By establishing a listening port to listen for requests to establish tunnels from other processes or containers. Other processes or containers can complete the establishment of the tunnel by connecting to this port and providing a packet identifier to this port, and an appropriate tunnel ID will be assigned.

[0043] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module also synchronizes tunnel information in the following manner:

[0044] Summarize all the established tunnels, group the tunnels using the packet identifiers of the tunnels, and sort the tunnels within the same group according to the tunnel ID.

[0045] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module also shares tunnel group information in the following manner:

[0046] After completing the synchronization of tunnel information to obtain tunnel group data, store the tunnel group data in the system cache and share the cache for other modules to use to achieve the sharing process of tunnel group information.

[0047] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module collects mirror traffic through DPDK and collects in-cloud traffic through cloud probe software.

[0048] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module performs traffic parsing processing in the following manner:

[0049] After completing the traffic collection process, if the obtained data packet is a three-layer or higher protocol data packet, then parse the source IP and destination IP of the data packet; if the obtained data packet is a two-layer protocol data packet, then parse the source MAC and destination MAC of the data packet.

[0050] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module performs traffic distribution processing in the following manner:

[0051] When the parsed data packet is the source IP and the destination IP, compare the two in terms of size, and compare the smaller-valued IP with the distribution cache table to determine whether the IP is in the distribution cache table. If so, directly distribute the traffic to the corresponding tunnel according to the distribution cache table. Otherwise, send the IP to the distribution cache table update module to update the distribution cache table, and forward the data packet to the tunnel session corresponding to the tunnel ID in the updated distribution cache table according to the updated distribution cache table to implement traffic distribution processing.

[0052] As a preferred embodiment of the present invention, the traffic collection, parsing and distribution module performs traffic distribution processing in the following manner:

[0053] When the parsed data packet is the source MAC and the destination MAC, compare the two in terms of size, and compare the smaller-valued MAC with the distribution cache table to determine whether the MAC is in the distribution cache table. If so, directly forward the data packet to the tunnel session corresponding to the tunnel ID according to the distribution cache table. Otherwise, send the MAC to the distribution cache table update module to update the distribution cache table, and forward the data packet to the tunnel session corresponding to the tunnel ID in the updated distribution cache table according to the updated distribution cache table to implement traffic distribution processing.

[0054] As a preferred embodiment of the present invention, the distribution cache table update module specifically performs the following processing:

[0055] After receiving the IP or MAC input by the traffic collection, parsing and distribution module, first read the tunnel grouping data from the system cache, query the distribution cache table, correspond the tunnels in the distribution cache table with the tunnel data in the tunnel grouping data to obtain the usage times of each tunnel ID under each group, take the tunnel ID with the smallest usage times in each group, and write the corresponding IP or MAC and the tunnel ID into the distribution cache table. If there are multiple tunnel IDs with the smallest usage times in the group, take the one with the smallest ID value, and write the IP or MAC and the obtained tunnel ID into the distribution cache table together.

[0056] The following further illustrates the technical solution with specific embodiments:

[0057] The system for implementing traffic collection, aggregation and load distribution processing includes a distribution cache table, a traffic collection, parsing and distribution module, and a distribution cache table update module.

[0058] The described distribution cache table is a data table stored in the system cache that shows the correspondence between IP and tunnel ID, and between MAC and tunnel ID. The data format is JSON. The keys in the data are IP or MAC, and the values corresponding to the keys are lists composed of one or more tunnel IDs. Example data is as follows:

[0059] {"192.168.5.7":[4,6],"08:00:20:0A:8C:6D":[7]}

[0060] The described traffic collection, parsing, and distribution module includes a distribution tunnel, traffic collection, traffic parsing, and traffic distribution;

[0061] In a specific embodiment of the present invention, the distribution tunnel is as shown in steps 1-4 of Figure 1 :

[0062] As shown in step 1 of Figure 1 The traffic analysis process sends a request to establish a tunnel to the traffic collection, parsing, and distribution module and provides a packet identifier to the traffic collection, parsing, and distribution module;

[0063] After receiving the request, the traffic collection, parsing, and distribution module establishes a connection with the traffic analysis process and maintains the session of this connection as the tunnel for traffic distribution, and assigns a tunnel ID to this session;

[0064] As shown in steps 2-3 of Figure 1 The traffic collection, parsing, and distribution module merges the packet identifier and tunnel ID of the newly established tunnel with the existing tunnel packet information, synchronizes the tunnel packet information, and sorts the IDs under each packet to obtain tunnel packet data;

[0065] As shown in step 4 of Figure 1 The traffic collection, parsing, and distribution module stores the tunnel packet data in the system cache and shares the cache for other modules to use;

[0066] In a specific embodiment of the present invention, the traffic collection, traffic parsing, and traffic distribution are as shown in steps 1-6 of Figure 2 :

[0067] As shown in step 1 of Figure 2 By collecting mirror traffic and cloud probes collecting in-cloud traffic, and parsing the traffic data packets. If the data packet is a three-layer or higher protocol data packet, the source IP and destination IP of the data packet are parsed. If it is a two-layer protocol data packet, since the two-layer protocol data packet has no IP, the source MAC and destination MAC are parsed;

[0068] As shown in step Figure 2As shown in steps 2-3, compare the source IP and the destination IP, and take the IP with the smaller value.

[0069] As Figure 2 As shown in step 4, determine whether the IP is in the distribution cache table. If it is, forward the data packet to the tunnel session corresponding to the IP in the cache table; if not, send the IP to the distribution cache table update module.

[0070] As Figure 2 As shown in steps 2 and 5, compare the source MAC and the destination MAC, and take the MAC with the smaller value.

[0071] As Figure 2 As shown in step 6, determine whether the MAC is in the distribution cache table. If it is, forward the data packet to the tunnel session corresponding to the MAC in the cache table; if not, send the MAC to the distribution cache table update module.

[0072] In a specific embodiment of the present invention, the distribution cache table update module is as Figure 2 shown in steps 7-9:

[0073] As Figure 2 As shown in steps 7-8, after the distribution cache table update module receives the IP or MAC sent by the traffic collection, parsing and distribution module, it first reads the tunnel packet data from the cache, then queries the distribution cache table, corresponds the tunnels in the distribution cache table with the tunnel data in the tunnel packet data to obtain the usage times of each tunnel ID under each packet, takes the tunnel ID with the smallest usage times in each packet, and writes the IP address or MAC address corresponding to the tunnel ID into the distribution cache table. If there are multiple tunnel IDs with the smallest usage times in the packet, take the one with the smallest ID value, and write the IP or MAC and the obtained tunnel ID into the distribution cache table together, as follows:

[0074] {"192.168.5.7":[4,6],"08:00:20:0A:8C:6D":[7],"10.55.144.65":[5,9]}

[0075] As Figure 2 As shown in step 9, store the updated distribution cache table into the system memory, share it for other modules to use, and forward the collected data packets to the corresponding tunnel sessions based on the updated distribution cache table.

[0076] Any process or method description depicted in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations where functions may be executed in a manner other than shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0077] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0078] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0079] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, or the like.

[0080] In the description of this specification, the description referring to terms such as "an embodiment", "some embodiments", "example", "specific example", or "embodiment", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0081] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0082] By adopting the system for realizing traffic collection aggregation and load distribution processing of the present invention, it is possible to enable various types of analysis modules to perform analysis simultaneously within the same computing resource, and multiple copies of each type of analysis module can be started to jointly complete the work, so as to achieve the goal of doubling or several times improving the analysis efficiency within the same computing resource; and traffic data packets with the same IP or the same MAC will be distributed into the same analysis tunnel, which can ensure that the analysis module connected to the analysis tunnel obtains the complete traffic of the IP or the MAC, and the accuracy of the analysis will not be affected by the distribution of the traffic, making the analysis process no longer a bottleneck, so as to maximize the utilization of computing resources and achieve cost reduction and efficiency improvement for users.

[0083] In this specification, the present invention has been described with reference to its specific embodiments. However, it is obvious that various modifications and transformations can still be made without departing from the spirit and scope of the present invention. Therefore, the specification and the drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system for implementing traffic collection aggregation and load distribution processing, characterized in that The system described above includes the following components: A distribution cache table for recording the correspondence between IP addresses and tunnel IDs, as well as MAC addresses and tunnel IDs in the system cache. A traffic collection, parsing, and distribution module connected to the distribution cache table for performing traffic collection, parsing, and distribution processing by establishing distribution tunnels. Specifically: The traffic analysis process sends a request to establish a tunnel to the traffic collection, parsing, and distribution module and provides a packet identifier to the module. After receiving the request, the traffic collection, parsing, and distribution module establishes a connection with the traffic analysis process and maintains the session of this connection as a traffic distribution tunnel, and assigns a tunnel ID to this session. The traffic collection, parsing, and distribution module merges the packet identifier and tunnel ID of the newly established tunnel with the existing tunnel packet information, synchronizes the tunnel packet information, and sorts the IDs under each packet to obtain tunnel packet data. And A distribution cache table update module connected to the traffic collection, parsing, and distribution module for, after receiving the IP address or MAC address input by the traffic collection, parsing, and distribution module, storing the updated distribution cache table in the system memory, sharing it for use by other modules, and forwarding the collected data packets to the corresponding tunnel session based on the updated distribution cache table for use.

2. The system for realizing traffic collection aggregation and load distribution processing according to claim 1, wherein The specific description of the tunnel ID is as follows: After the establishment of the tunnel session is completed, a tunnel ID will be assigned to each tunnel.

3. The system for implementing traffic collection aggregation and load distribution processing according to claim 1, wherein The traffic collection, parsing, and distribution module establishes a distribution tunnel in the following way: By establishing a listening port to listen for requests to establish tunnels from other processes or containers. Other processes or containers complete the establishment of the tunnel by connecting to this port and providing a packet identifier to this port, and a corresponding tunnel ID is assigned.

4. The system for realizing traffic collection aggregation and load distribution processing according to claim 3, wherein The traffic collection, parsing, and distribution module also synchronizes tunnel information in the following way: Summarize all the established tunnels, group the tunnels using the packet identifiers of the tunnels, and sort the tunnels within the same group according to the tunnel ID.

5. The system for implementing traffic collection aggregation and load distribution processing according to claim 4, characterized in that The traffic collection, parsing, and distribution module also shares tunnel packet information in the following way: After completing the synchronization of tunnel information to obtain tunnel packet data, store the tunnel packet data in the system cache and share the cache for use by other modules to achieve the sharing process of tunnel packet information.

6. The system for implementing traffic collection aggregation and load distribution processing according to any one of claims 1 to 5, characterized in that, The traffic collection, parsing, and distribution module collects mirror traffic through DPDK and collects in-cloud traffic through cloud probe software.

7. The system for implementing traffic collection aggregation and load distribution processing according to claim 6, wherein The traffic collection, parsing, and distribution module performs traffic parsing processing in the following way: After completing the traffic collection processing, if the obtained data packet is a three-layer or higher protocol data packet, the source IP and destination IP of the data packet are parsed; if the obtained data packet is a two-layer protocol data packet, the source MAC and destination MAC of the data packet are parsed.

8. The system for implementing traffic collection aggregation and load distribution processing according to claim 7, wherein, The traffic collection, parsing, and distribution module performs traffic distribution processing in the following way: When the parsed data packet is the source IP and the destination IP, compare the two in terms of size, and compare the smaller-valued IP with the distribution cache table to determine whether the IP is in the distribution cache table. If so, directly distribute traffic to the corresponding tunnel according to the distribution cache table; otherwise, send the IP to the distribution cache table update module to update the distribution cache table, and forward the data packet to the tunnel session corresponding to the tunnel ID in the updated distribution cache table according to the updated distribution cache table to implement traffic distribution processing.

9. The system for implementing traffic collection aggregation and load distribution processing according to claim 7, characterized in that, The traffic collection, parsing and distribution module performs traffic distribution processing in the following manner: When the parsed data packet is the source MAC and the destination MAC, compare the two in terms of size, and compare the smaller-valued MAC with the distribution cache table to determine whether the MAC is in the distribution cache table. If so, directly forward the data packet to the tunnel session corresponding to the tunnel ID according to the distribution cache table; otherwise, send the MAC to the distribution cache table update module to update the distribution cache table, and forward the data packet to the tunnel session corresponding to the tunnel ID in the updated distribution cache table according to the updated distribution cache table to implement traffic distribution processing.

10. The system for implementing traffic collection aggregation and load distribution processing according to claim 8 or 9, characterized in that, The distribution cache table update module specifically performs the following processing: After receiving the IP or MAC input by the traffic collection, parsing and distribution module, first read the tunnel grouping data from the system cache, query the distribution cache table, correspond the tunnels in the distribution cache table with the tunnel data in the tunnel grouping data to obtain the usage times of each tunnel ID under each grouping, select the tunnel ID with the smallest usage times in each grouping, and write the IP or MAC corresponding to this tunnel ID into the distribution cache table. If there are multiple tunnel IDs with the smallest usage times in the grouping, select the one with the smallest ID value among them, and write the IP or MAC together with the obtained tunnel ID into the distribution cache table.

Images