Flow mirroring method and system based on bloom filter

Abstract

The invention discloses a traffic mirroring method and system based on a Bloom filter, and the method comprises the following steps: S1, network equipment copies original traffic to generate mirroring traffic; s2, judging whether to start the Bloom filter or not according to configuration requirements, and if yes, executing S3; if not, executing S4; s3, executing Bloom filter check on the mirror image traffic; s4, performing traditional sampling mirroring on the mirroring traffic passing through the Bloom filter according to a preset sampling ratio to obtain a final mirroring traffic message; and S5, outputting a final mirror image message and monitoring equipment. According to the method, the Bloom filter is introduced into the mirroring, and mirroring can be performed according to the message characteristics, so that the mouse flow can be ensured not to be ignored, and the problem of impact of the elephant flow on the network can be solved. The method provided by the invention has relatively strong practical application significance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a flow mirroring method and system based on a Bloom filter. Background Technology

[0002] Port mirroring Port mirroring technology can copy packets from a specified source to a destination port without interfering with the normal packet processing flow of network devices such as switches and routers. The destination port is connected to monitoring equipment directly or indirectly, and the analysis software on the monitoring equipment can analyze the copied packets. When the network is attacked or malfunctions, network administrators can use the mirroring function to obtain packets and perform in-depth analysis to locate the source of the attack or troubleshoot the cause of the fault. This technology, through a packet copying mechanism independent of the main forwarding path, ensures that network performance is not affected while providing critical data support for network security monitoring and maintenance. Figure 1 (As shown) Bloom filters The Bloom filter, proposed by Bloom in 1970, is essentially a long binary vector and a series of random mapping functions. Bloom filters can be used to check whether an element is in a set. Its advantages are significantly better space efficiency and query time than general algorithms; its disadvantages include a certain false positive rate and difficulty in deletion.

[0003] A Bloom filter is essentially composed of a bit vector or bit list of length n (a list containing only 0 or 1 bit values), with all values ​​initially set to 0, as shown in the table below.

[0004]

[0005] Table 1 like Figure 2 As shown, when `pkt info` is input, the corresponding index positions are set to 1 based on the outputs of k hash functions (a, b, c). When searching for `pkt info`, if any index position after the k hash functions is 0, then the value is definitely not in the set. However, if all hash index values ​​are 1, then the value may be in the set.

[0006] Deficiencies of existing technology With the rapid development of technologies such as 5G, IoT, big data, cloud computing, and AI, network data traffic is experiencing explosive growth. Traditional mirroring technology supports sampling and mirroring traffic at a fixed sampling ratio (e.g., 100:1), but it faces the following challenges in complex network environments: Elephant Stream (High Traffic Dominance): The mirrored packets generated by elephant streams may overwhelm the monitoring system. A large number of mirrored packets may consume too much network bandwidth and storage resources, exacerbating the network burden.

[0007] Mouse flow (low flow) missed detection: Under a fixed sampling ratio, low flow packets are easily ignored due to their low sampling probability and cannot be captured in time.

[0008] like Figure 3 As shown (green represents elephant flow, purple represents mouse flow), the mouse flow (small flow) is ignored under a fixed sampling ratio of 100:1. Summary of the Invention

[0009] According to a first aspect of the present invention, a flow mirroring method based on a Bloom filter is provided, comprising the following steps: S1: The network device copies the original traffic to generate mirrored traffic; S2: Based on configuration requirements, determine whether to enable the Bloom filter. If enabled, proceed to S3; otherwise, proceed to S4. S3: Perform a Bloom filter check on the mirrored traffic; S4: Perform traditional sampling mirroring on the mirrored traffic passing through the Bloom filter according to the preset sampling ratio to obtain the final mirrored traffic message; S5: Output the final mirror message to the monitoring device.

[0010] Furthermore, the specific steps for performing a Bloom filter check on the mirrored traffic are as follows: Extract the characteristic information of the mirrored traffic packets and calculate the bitmap index using "k" hash functions; Check the value of the corresponding index bit in the bitmap: If any index bit is 0, the packet is determined to be unmirrored, a mirroring operation is performed, and all index bits are set to 1. If all index bits are 1, discard the mirrored message.

[0011] Furthermore, when performing Bloom filter checks on mirrored traffic, a timing mechanism is introduced to clear the bitmap through a timer period, which is configurable by the user.

[0012] Furthermore, the default reset period of the timer is 1 second, and users can dynamically adjust the period length.

[0013] Furthermore, traditional sampling mirroring uses a fixed sampling ratio, with a sampling ratio ranging from 1:N, where N is an integer ≥1.

[0014] Furthermore, the Bloom filter and traditional sampling mirror can be enabled independently or in combination, including three operating modes: Mode 1: Enable Bloom filters only; Mode 2: Only enable traditional sampling mirroring; Mode 3: Enable both Bloom filters and traditional sampling mirroring simultaneously.

[0015] According to a second aspect of the present invention, a flow mirroring system based on a Bloom filter is provided, comprising: Traffic replication module: used to replicate raw traffic to generate mirrored traffic; Bloom filter module: configured for Bloom filter check operation of the first aspect; Traditional sampling module: configured to sample mirrored traffic at a preset ratio; Control module: Dynamically configures the activation status and parameters of the Bloom filter module and the traditional sampling module.

[0016] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor executes the computer program to perform a flow mirroring method based on a Bloom filter according to the first aspect.

[0017] According to a fourth aspect of the present invention, a readable medium having processor-executable non-volatile program code is provided, the program code causing the processor to run a flow mirroring method based on a Bloom filter according to the first aspect.

[0018] According to an embodiment of the present invention, a traffic mirroring method and system based on a Bloom filter can perform mirroring based on packet characteristics by introducing a Bloom filter into the mirroring process. This ensures that "mouse flows" are not ignored while mitigating the impact of "elephant flows" on the network. This method has significant practical application value.

[0019] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating port mirroring.

[0021] Figure 2 This is a schematic diagram of a Bloom filter.

[0022] Figure 3 This is a schematic diagram of a 100:1 mirror image sampling.

[0023] Figure 4 This is a schematic diagram of a Bloom filter in a flow mirroring method based on a Bloom filter according to an embodiment of the present invention.

[0024] Figure 5This is a schematic diagram illustrating a flow mirroring method based on a Bloom filter according to an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram of a flow mirroring method based on a Bloom filter according to an embodiment of the present invention, which includes two types of filter mirroring.

[0026] Figure 7 This invention relates to a flow mirroring method based on a Bloom filter, which describes the use of two filters.

[0027] Figure 8 This is a flowchart of a flow mirroring method based on a Bloom filter according to an embodiment of the present invention.

[0028] Figure 9 This is a structural diagram of a flow mirroring system based on a Bloom filter according to an embodiment of the present invention.

[0029] Figure 10 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0030] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, further illustrating the present invention.

[0031] First, combine Figures 4-8 This invention describes a flow mirroring method based on a Bloom filter, which is used for flow mirroring and has a wide range of applications.

[0032] like Figures 4-8 As shown, an embodiment of the present invention provides a flow mirroring method based on a Bloom filter, comprising the following steps: S1: The network device copies the original traffic to generate mirrored traffic.

[0033] S2: Based on configuration requirements, determine whether to enable the Bloom filter. If enabled, proceed to S3; otherwise, proceed to S4. S3: Perform a Bloom filter check on the mirrored traffic. Specifically, performing a Bloom filter check on the mirrored traffic involves: Extract the characteristic information of the mirrored traffic packets and calculate the bitmap index using "k" hash functions; Check the value of the corresponding index bit in the bitmap: If any index bit is 0, the packet is determined to be unmirrored, a mirroring operation is performed, and all index bits are set to 1. If all index bits are 1, it means that the traffic has been mirrored. Discard the mirrored packet, and eventually no more mirrored traffic will be generated.

[0034] Furthermore, in this embodiment, when performing Bloom filter checks on mirrored traffic, a timing mechanism is introduced to ensure the continuous generation of mirrored traffic. By periodically clearing the bitmap using a timer, all relevant index positions of the Bloom filter are reset to 0. This allows mirrored packets generated in the previous cycle to be generated again after another time cycle. The clearing cycle is configurable by the user. The default clearing cycle of the timer is 1 second, and the user can dynamically adjust the cycle length.

[0035] S4: For the mirrored traffic passing through the Bloom filter, perform traditional sampling mirroring according to a preset sampling ratio to obtain the final mirrored traffic packets. Traditional sampling mirroring uses a fixed sampling ratio, ranging from 1:N, where N is an integer ≥ 1.

[0036] S5: Output the final mirror message to the monitoring device.

[0037] It should be noted that, as Figure 5 As shown, when mirroring packets, the system first checks if a Bloom filter is enabled based on the configuration. If enabled, the mirrored packets will be checked by the Bloom filter; otherwise, the mirrored packets will proceed according to the normal process. However, even with only a Bloom filter, there's still a possibility of large "elephant flows" impacting the network. In this case, traditional sampling ratio methods are still needed to rate-limit the mirrored packets. Therefore, a complete mirroring sampling scheme combines Bloom filtering and traditional sampling methods, such as... Figure 6 As shown. Furthermore, these two types of filters can be used together or separately, such as... Figure 7 As shown.

[0038] Furthermore, in this embodiment, the Bloom filter and the traditional sampling mirror can be enabled independently or in combination, including three operating modes: Mode 1: Enable Bloom filters only; Mode 2: Only enable traditional sampling mirroring; Mode 3: Enable both Bloom filters and traditional sampling mirroring simultaneously.

[0039] As described above, in a traffic mirroring method based on a Bloom filter according to an embodiment of the present invention, by introducing a Bloom filter into the mirroring process, mirroring can be performed according to packet characteristics. This ensures that "mouse flows" are not ignored while also addressing the impact of "elephant flows" on the network. This method has significant practical application value.

[0040] The above combined with the appendix Figures 4-8 A flow mirroring method based on a Bloom filter according to an embodiment of the present invention is described. Furthermore, the present invention can also be applied to a flow mirroring system based on a Bloom filter.

[0041] like Figure 9 As shown, according to a second aspect of the present invention, a flow mirroring system based on a Bloom filter is provided, comprising: Traffic replication module 100: Used to replicate raw traffic to generate mirrored traffic; Bloom filter module 200: configured for Bloom filter check operation of the first aspect; Traditional sampling module 300: configured to sample mirrored traffic at a preset ratio; Control module 400: Dynamically configures the activation status and parameters of Bloom filter module 200 and conventional sampling module 300.

[0042] The above combined with the appendix Figure 9 A flow mirroring system based on a Bloom filter according to an embodiment of the present invention is described. Furthermore, the present invention can also be applied to an electronic device.

[0043] like Figure 10 As shown, according to a third aspect of the present invention, an electronic device is provided, comprising: a memory 1, a processor 2, and a computer program 3, wherein the computer program 3 is stored in the memory 1, and the processor 2 executes the computer program 3 to perform a flow mirroring method based on a Bloom filter according to the first aspect.

[0044] According to a fourth aspect of the present invention, a readable medium having processor-executable non-volatile program code is provided, the program code causing the processor to run a flow mirroring method based on a Bloom filter according to the first aspect.

[0045] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of a computer program from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the ASIC can reside within a device. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. The present invention also provides a program product comprising executable instructions stored in the readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the execution of the executable instructions by the at least one processor causes the device to implement a Bloom filter-based flow mirroring method provided in the various embodiments described above. In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0046] It should be noted that, in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0047] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A flow mirroring method based on a Bloom filter, characterized in that, It includes the following steps: S1: The network device copies the original traffic to generate mirrored traffic; S2: Based on configuration requirements, determine whether to enable the Bloom filter. If enabled, proceed to S3; otherwise, proceed to S4. S3: Perform a Bloom filter check on the mirrored traffic; S4: Perform traditional sampling mirroring on the mirrored traffic passing through the Bloom filter according to the preset sampling ratio to obtain the final mirrored traffic message; S5: Output the final mirror message to the monitoring device.

2. The flow mirroring method based on a Bloom filter as described in claim 1, characterized in that, The specific steps of performing a Bloom filter check on the mirrored traffic are as follows: Extract the characteristic information of the mirrored traffic packets and calculate the bitmap index using "k" hash functions; Check the value of the corresponding index bit in the bitmap: If any index bit is 0, the packet is determined to be unmirrored, a mirroring operation is performed, and all index bits are set to 1. If all index bits are 1, discard the mirrored message.

3. The flow mirroring method based on a Bloom filter as described in claim 2, characterized in that, When performing a Bloom filter check on the mirrored traffic, a timing mechanism is introduced, which uses a timer period to clear the bitmap. The clearing period is configurable by the user.

4. The flow mirroring method based on a Bloom filter as described in claim 3, characterized in that, The default reset period of the timer is 1 second, and the user can dynamically adjust the period length.

5. The flow mirroring method based on a Bloom filter as described in claim 1, characterized in that, The conventional sampling mirror uses a fixed sampling ratio, with a sampling ratio ranging from 1:N, where N is an integer ≥1.

6. The flow mirroring method based on a Bloom filter as described in claim 1, characterized in that, The Bloom filter and the conventional sampling mirror can be enabled independently or in combination, including three operating modes: Mode 1: Enable Bloom filters only; Mode 2: Only enable traditional sampling mirroring; Mode 3: Enable both Bloom filters and traditional sampling mirroring simultaneously.

7. A flow mirroring system based on a Bloom filter, characterized in that, Include: Traffic replication module: used to replicate raw traffic to generate mirrored traffic; Bloom filter module: configured to perform the Bloom filter inspection operation as described in any one of claims 1-6; Traditional sampling module: configured to sample mirrored traffic at a preset ratio; Control module: Dynamically configures the activation status and parameters of the Bloom filter module and the traditional sampling module.

8. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program, the computer program being stored in the memory, and the processor executing the computer program to perform a flow mirroring method based on a Bloom filter as described in any one of claims 1 to 6.

9. A readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute a flow mirroring method based on a Bloom filter as described in any one of claims 1-6.

Images