Bit map-based full-flow forwarding distributed deduplication method and related equipment
By managing network session states using the Bitmap algorithm, the problems caused by duplicate forwarding and abnormal centralized coordinator in distributed network traffic collection systems are solved, achieving efficient network session deduplication and throughput improvement.
Patent Information
- Application Number
- CN202511926674.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-03-13
AI Technical Summary
In distributed network traffic collection and forwarding systems, the lack of effective session deduplication control leads to network bandwidth waste, system resource redundancy, and data consistency issues, as well as session loss and reduced throughput when the centralized coordinator malfunctions.
A distributed deduplication method based on bitmap is adopted. The network session state is recorded and managed through the bitmap algorithm. The session existence judgment is realized in O(1) time complexity by using bit operations of bitmap to avoid repeated forwarding. The session information between nodes is synchronized through the controller, and the bitmap is dynamically managed to reduce the amount of state synchronization data.
It achieves efficient network session deduplication, reduces network overhead, supports large-scale network session management, ensures the real-time and accuracy of session forwarding, and avoids delays caused by duplicate forwarding and centralized coordinators.
Smart Images
Figure CN121664867A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of packet forwarding, and specifically to a method and related equipment for distributed deduplication of full-traffic forwarding based on bitmap. Background Technology
[0002] In distributed network traffic collection and forwarding systems, multiple collection nodes (such as gateways, proxies, and edge nodes) may simultaneously receive traffic data from the same network session (such as the same TCP / UDP connection, 5-tuple stream, etc.). If these nodes do not perform effective session deduplication control, the traffic of the same session may be repeatedly forwarded by multiple nodes, resulting in wasted network bandwidth, redundant system resources, and potential data consistency issues. To solve this problem, traditional solutions typically adopt a centralized session state management approach, which introduces a centralized coordinator, such as Redis or ZooKeeper. Before processing a session, all nodes first query the analysis center service to see if the session has already been forwarded. If not, forwarding is allowed, and the network session is recorded in the analysis center; if it already exists, it is skipped or abandoned.
[0003] Currently, traditional solutions have the following technical drawbacks: 1. If any forwarding node abnormally exits, other nodes cannot detect its offline status, resulting in the loss of all network sessions of the offline node, which will no longer be forwarded by other nodes. 2. Each forwarding requires querying the analysis center, introducing additional latency and reducing throughput. Summary of the Invention
[0004] To address the aforementioned shortcomings of existing technologies, this invention provides a method and related equipment for distributed deduplication of full-traffic forwarding based on bitmaps. This invention can effectively solve the problems of insufficient consistency and throughput in network session forwarding of traditional technologies.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A first aspect of the present invention provides a method for distributed deduplication of full-traffic forwarding based on bitmaps, the method comprising:
[0007] Record the first network session that the target acquisition node is currently forwarding, and the second network session that other acquisition nodes in the acquisition node cluster are forwarding;
[0008] Calculate the session IDs for the first network session and the second network session respectively; where the session ID is a unique 64-bit number.
[0009] Based on the session IDs of the first network session and the second network session, determine whether the session ID of the first network session exists in the cluster bitmap of the acquisition node cluster. If it exists, the target acquisition node abandons the first network session that is currently being forwarded; otherwise, insert the session ID of the first network session into the node bitmap of the target acquisition node and perform the forwarding operation.
[0010] In one implementation, calculating the session IDs of the first network session and the second network session includes:
[0011] The first and second network sessions are parsed to identify their IP types.
[0012] The session ID is calculated based on the IP type of the first and second network sessions.
[0013] In one implementation, the session ID is calculated based on the IP type of the first network session and the second network session, including:
[0014] If both the first network session and the second network session have IPv4 IPs, then the IP addresses of the first network session and the second network session are sorted and concatenated to obtain the session IDs of the first network session and the second network session.
[0015] If both the first network session and the second network session have IPv6 IPs, then the IP addresses of the first network session and the second network session are sorted and hashed to obtain the session IDs of the first network session and the second network session.
[0016] In one implementation, the FNV hash algorithm is used for hashing.
[0017] In one implementation, the method further includes:
[0018] Periodically receive the node bitmap sent by the target acquisition node;
[0019] Perform an OR operation on the node bitmaps of all acquisition nodes except the target acquisition node to generate the cluster bitmap of the target acquisition node;
[0020] By performing a bitwise AND operation on the cluster bitmap and node bitmap of the target acquisition node, a collision detection is performed to obtain a judgment result on whether there are conflicting bits.
[0021] Based on the judgment result, perform the corresponding operation; the operation includes issuing a pruning command or storing the node bitmap.
[0022] In one implementation, the corresponding operation is performed based on the judgment result, including:
[0023] If a conflicting bit is detected, a pruning command is sent to the target acquisition node and the node bit map sent by the target acquisition node is discarded. The target acquisition node responds to the pruning command by pruning the conflicting bits in its own node bit map and updating the cluster bit map of the target acquisition node.
[0024] If no conflicting bits are detected, store the node bit map of the target acquisition node.
[0025] In one implementation, the method further includes: if no node bitmap sent by the target acquisition node is received when the reporting time reaches the threshold, the node bitmap of the target acquisition node is removed from memory and broadcast to other acquisition nodes, and the other acquisition nodes remove the broadcast bits from the cluster bitmap.
[0026] A second aspect of the present invention provides an electronic device, including a memory and a processor;
[0027] A memory for storing computer programs, the computer programs including program instructions;
[0028] A processor is configured to execute the program instructions to cause the electronic device to perform the steps of a bitmap-based full-traffic forwarding distributed deduplication method as provided in the first aspect of the invention.
[0029] A third aspect of the present invention provides a computer program product comprising program instructions that, when executed by an electronic device, cause the electronic device to perform the steps of a bitmap-based full-traffic forwarding distributed deduplication method provided in the first aspect of the present invention.
[0030] A fourth aspect of the present invention provides a computer-readable storage medium comprising a computer program that, when executed by one or more processors, implements the steps of a bitmap-based full-traffic forwarding distributed deduplication method as provided in the first aspect of the present invention.
[0031] The technical solution provided by this invention has the following advantages compared with the known prior art:
[0032] 1. This invention utilizes bitmaps to achieve O(1) time complexity for accurate session existence determination and avoids duplicate forwarding, thereby achieving efficient deduplication of network sessions.
[0033] 2. Compared with the synchronous and complete session table provided by traditional technology, the present invention uses a bitmap to greatly reduce the amount of data for state synchronization, significantly reduce network overhead, and save network bandwidth.
[0034] 3. Each network session in this invention occupies only 1 bit, supporting efficient management of a large number of network sessions. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0036] Figure 1 A schematic diagram of a prior art distributed network traffic collection and forwarding system;
[0037] Figure 2 This is a schematic diagram of a distributed network traffic collection and forwarding system provided in an embodiment of the present invention;
[0038] Figure 3 A flowchart illustrating a distributed deduplication method for full-traffic forwarding based on bitmap provided in an embodiment of the present invention;
[0039] Figure 4 A schematic diagram illustrating the calculation of a session ID of IPv4 type provided in an embodiment of the present invention;
[0040] Figure 5 A schematic diagram illustrating the calculation of a session ID of IPv6 type provided in an embodiment of the present invention;
[0041] Figure 6 This is a schematic diagram illustrating the management of a distributed bitmap provided in an embodiment of the present invention;
[0042] Figure 7 This is a schematic diagram illustrating the coordination of the central node and the synchronization of the cluster bit map according to an embodiment of the present invention. Detailed Implementation
[0043] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0044] In distributed network traffic collection and forwarding systems, multiple collection nodes (such as gateways, proxies, and edge nodes) may simultaneously receive traffic data from the same network session (such as the same TCP / UDP connection, a five-tuple stream, etc.). If effective session deduplication control is not performed among these nodes, traffic from the same session may be repeatedly forwarded by multiple nodes, resulting in wasted network bandwidth, system resource redundancy, and potential data consistency issues. To solve this problem, traditional solutions typically employ centralized session state management, such as... Figure 1As shown, a centralized coordinator, such as Redis or ZooKeeper, is introduced. Before processing a session, all nodes query the analysis center service to see if the session has already been forwarded. If not, forwarding is allowed, and the network session is recorded in the analysis center. If the session already exists, it is skipped or abandoned.
[0045] Currently, traditional solutions have the following technical drawbacks: 1. If any forwarding node abnormally exits, other nodes cannot detect its offline status, resulting in the loss of all network sessions of the offline node, which will no longer be forwarded by other nodes. 2. Each forwarding requires querying the analysis center, introducing additional latency and reducing throughput.
[0046] To address the aforementioned problems, this invention proposes a distributed deduplication method based on bitmap-based full-traffic forwarding, such as... Figure 2 As shown, this invention introduces a controller for synchronizing network session information between nodes, and the controller is responsible for maintaining the bitmap of all acquisition nodes in the environment.
[0047] The core of the roaring bitmap algorithm is to use a bit array to record two states, 0 and 1, and then map the specific data to a specific position in this bit array. Setting the bit to 0 indicates that the data does not exist, and setting it to 1 indicates that the data exists. Table 1 below shows a bitmap of length 16, whose initial state is all 0s.
[0048] Table 1
[0049]
[0050] When the number 6 is inserted into the bitmap, the bitmap sets the index 6 to 1, as shown in Table 2 below:
[0051] Table 2
[0052]
[0053] Since it only has 16 bits, the existence of the number 6 in the bitmap can be directly determined by a single AND operation, i.e., bitmap & (1<<6) != 0. In order to take advantage of this feature of the bitmap to achieve efficient deduplication and save memory overhead, this invention relies on the roaring bitmap algorithm to dynamically manage the bitmap.
[0054] like Figure 3 As shown, this embodiment of the invention provides a distributed deduplication method for full-traffic forwarding based on bitmaps, the method comprising:
[0055] S301 records the first network session that the target acquisition node is currently forwarding, and the second network session that other acquisition nodes in the acquisition node cluster are forwarding.
[0056] Specifically, the operation of recording network sessions is performed using the publicly available Roaring Bitmap algorithm. In short, it involves using a shift operation in an infinitely long bitmap to record whether a uint64-bit number exists. For specific implementation details, please refer to the publicly available Roaring Bitmap algorithm.
[0057] S302, calculate the session IDs for the first network session and the second network session respectively; where the session ID is a unique 64-bit number.
[0058] Specifically, a network session identifier is generally composed of a five-tuple, such as source IP, destination IP, protocol, source port, and destination port.
[0059] In this embodiment, a tuple is selected to calculate the session ID. A tuple is a general concept in the TCP / IP protocol, which refers to the source IP address and destination IP address of the two communicating parties.
[0060] In this embodiment, the first network session and the second network session are first parsed to identify the IP type of the first network session and the second network session, such as IPv4 and IPv6.
[0061] Then, the session ID is calculated based on the IP type of the first and second network sessions. For example, such as... Figure 4 As shown, if the IP types of the first network session and the second network session are both IPv4, then the IP addresses of the first network session and the second network session are sorted and concatenated to obtain the session IDs of the first network session and the second network session.
[0062] Specifically, IPv4 is only 32 bits long in memory, so it can be directly represented as a 32-bit integer. Then, the 32-bit integers converted from source and destination IPs are sorted, the smaller one is left-shifted by 32 bits and ORed with the larger one to generate a 64-bit integer, which is the IPv4 session ID.
[0063] like Figure 5 As shown, if the IP types of the first network session and the second network session are both IPv6, then the IP addresses of the first network session and the second network session are sorted and hashed to obtain the session IDs of the first network session and the second network session.
[0064] Specifically, IPv6 is 128 bits long in memory, and the source and destination IPs together are 256 bits. This cannot be directly converted to a 64-bit number. Therefore, the source and destination IPs are sorted by ASCII and concatenated into a 256-bit / 32-byte array. Then, a bitwise FNV hash operation is performed to obtain the 64-bit IPv6 session ID.
[0065] Specifically, the FNV hash algorithm is used for hashing. The FNV hash algorithm is a publicly available, universal algorithm that can convert input data of any length into a 64-bit number. Here, two IPv6 data entries are concatenated into a 256-bit data set, input as 8 bytes, and then the FNV operation is used to obtain a 64-bit output.
[0066] S303, determine whether the session ID of the first network session exists in the cluster bit map of the collection node cluster based on the session ID of the first network session and the second network session. If it exists, the target collection node abandons the first network session that is currently being forwarded. Otherwise, insert the session ID of the first network session into the node bit map of the target collection node and perform the forwarding operation.
[0067] Specifically, to determine whether a session ID exists in the cluster bitmap of the acquisition node cluster, this embodiment uses the publicly available Roaring Bitmap algorithm. In short, the bitmap maintains multiple 64-bit masks in an array. The session ID is used to divide the array length integer to find the array index, and then modulo 64 is taken to obtain the mask bit. The existence of the ID is determined by checking if this bit is 1. In practical applications, there are also some optimization steps, making it more complex; details can be found in the description of the Roaring Bitmap.
[0068] like Figure 6 As shown, if the session ID of the first network session exists in the cluster bitmap of the collection node cluster, it means that the first network session has been processed by other collection nodes, so the target collection node abandons the network session that is being forwarded.
[0069] Assuming a single node maintains approximately 2 million sessions, leveraging the characteristics of the roaring bitmap algorithm, the corresponding bitmap occupies between 8k and 20M of memory. Compared to traditional session table synchronization methods (such as transmitting complete connection state information), this significantly reduces network bandwidth consumption. Through bit operations on the bitmap, high-performance session state judgment and updates can be achieved, while ensuring the real-time performance and accuracy of deduplication logic.
[0070] Based on the distributed bitmap management mechanism described in the above embodiments, the method provided in this embodiment of the invention further includes: periodically receiving node bitmaps sent by the target acquisition node; performing OR operations on the node bitmaps of all acquisition nodes other than the target acquisition node to generate a cluster bitmap of the target acquisition node; performing conflict detection on the cluster bitmap and node bitmap of the target acquisition node through bitwise AND operations to obtain a judgment result on whether conflicting bits exist; and performing corresponding operations based on the judgment result; wherein, the operations include issuing pruning instructions or storing node bitmaps.
[0071] Specifically, such as Figure 7 As shown, the corresponding operations are performed based on the judgment result, including: if a conflicting bit is detected, a pruning instruction is sent to the target acquisition node and the node bit map sent by the target acquisition node is discarded. The target acquisition node responds to the pruning instruction by pruning the conflicting bits in its own node bit map and updating the cluster bit map of the target acquisition node; if no conflicting bit is detected, the node bit map of the target acquisition node is stored.
[0072] In some embodiments, the method further includes: if no node bitmap sent by the target acquisition node is received when the reporting time reaches a threshold, the node bitmap of the target acquisition node is removed from memory and broadcast to other acquisition nodes, and the other acquisition nodes remove the broadcast bits from the cluster bitmap.
[0073] Specifically, when a data collection node goes offline (e.g., it has not reported bitmap information for a long time), it is determined that it is no longer active. The bitmap of that data collection node is removed from memory and broadcast to other nodes. The other nodes then remove the broadcast bitmap from the cluster bitmap.
[0074] This invention also provides an electronic device. The electronic device includes a processor, a memory, a communication interface, and at least one communication bus for connecting the processor, the memory, and the communication interface. The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (PROM), or portable read-only memory (CD-ROM), and is used for related instructions and data.
[0075] The communication interface is used to receive and send data. The processor can be one or more CPUs; if the processor is a single CPU, it can be a single-core CPU or a multi-core CPU. The processor in the electronic device reads one or more programs stored in memory and performs the following operations: records the first network session currently being forwarded by the target acquisition node, and the second network session currently being forwarded by other acquisition nodes in the acquisition node cluster; calculates the session IDs of the first and second network sessions respectively; where the session ID is a unique 64-bit number; determines whether the session ID of the first network session exists in the cluster bitmap of the acquisition node cluster based on the session IDs of the first and second network sessions; if it exists, the target acquisition node abandons the currently forwarded first network session; otherwise, it inserts the session ID of the first network session into the node bitmap of the target acquisition node and performs the forwarding operation.
[0076] It should be noted that the specific implementation of each operation can be described above. Figure 3 The corresponding description of the method embodiments shown indicates that the electronic device can be used to execute a bitmap-based full-traffic forwarding distributed deduplication method according to the above method embodiments of this application, which will not be described in detail here.
[0077] This invention also provides a computer-readable storage medium, which is a memory device in a computer device for storing programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the bitmap-based full-flow forwarding distributed deduplication method in the above embodiments. Those skilled in the art should understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD ROM, optical storage, etc.) containing computer-usable program code.
[0078] This invention also provides a computer program product containing program instructions. The computer program product may be software or program products containing program instructions, capable of running on a computing device or stored on any available medium. When the computer program product runs on at least one electronic device, it causes the at least one electronic device to perform a bitmap-based full-flow forwarding distributed deduplication method.
[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A distributed deduplication method for full-traffic forwarding based on bitmaps, characterized in that, The methods include: Record the first network session that the target acquisition node is currently forwarding, and the second network session that other acquisition nodes in the acquisition node cluster are forwarding; Calculate the session IDs for the first network session and the second network session respectively; where the session ID is a unique 64-bit number. Based on the session IDs of the first network session and the second network session, determine whether the session ID of the first network session exists in the cluster bitmap of the acquisition node cluster. If it exists, the target acquisition node abandons the first network session that is currently being forwarded; otherwise, insert the session ID of the first network session into the node bitmap of the target acquisition node and perform the forwarding operation.
2. The method according to claim 1, characterized in that, Calculate the session IDs for the first network session and the second network session, including: The first and second network sessions are parsed to identify their IP types. The session ID is calculated based on the IP type of the first and second network sessions.
3. The method according to claim 1, characterized in that, Based on the IP type of the first network session and the second network session, calculate the session ID, including: If both the first network session and the second network session have IPv4 IPs, then the IP addresses of the first network session and the second network session are sorted and concatenated to obtain the session IDs of the first network session and the second network session. If both the first network session and the second network session have IPv6 IPs, then the IP addresses of the first network session and the second network session are sorted and hashed to obtain the session IDs of the first network session and the second network session.
4. The method according to claim 3, characterized in that, The FNV hash algorithm is used for hash calculation.
5. The method according to claim 1, characterized in that, The method further includes: Periodically receive the node bitmap sent by the target acquisition node; Perform an OR operation on the node bitmaps of all acquisition nodes except the target acquisition node to generate the cluster bitmap of the target acquisition node; By performing a bitwise AND operation on the cluster bitmap and node bitmap of the target acquisition node, a collision detection is performed to obtain a judgment result on whether there are conflicting bits. Based on the judgment result, perform the corresponding operation; the operation includes issuing a pruning command or storing the node bitmap.
6. The method according to claim 5, characterized in that, Based on the judgment result, perform the corresponding operations, including: If a conflicting bit is detected, a pruning command is sent to the target acquisition node and the node bit map sent by the target acquisition node is discarded. The target acquisition node responds to the pruning command by pruning the conflicting bits in its own node bit map and updating the cluster bit map of the target acquisition node. If no conflicting bits are detected, store the node bit map of the target acquisition node.
7. The method according to claim 5, characterized in that, The method further includes: if no node bitmap sent by the target acquisition node is received when the reporting time reaches the threshold, the node bitmap of the target acquisition node is removed from memory and broadcast to other acquisition nodes, and the other acquisition nodes remove the broadcast bits from the cluster bitmap.
8. An electronic device, characterized in that, Including memory and processor; A memory for storing computer programs, the computer programs including program instructions; A processor is configured to execute the program instructions to cause the electronic device to perform the steps of a bitmap-based full-traffic forwarding distributed deduplication method as described in any one of claims 1 to 7.
9. A computer program product containing program instructions, characterized in that, When the program instructions are executed by the electronic device, the electronic device performs the steps of a bitmap-based full-traffic forwarding distributed deduplication method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed by one or more processors, implements the steps of a bitmap-based full-traffic forwarding distributed deduplication method as described in any one of claims 1 to 7.