A method and system for measuring flow cardinality based on a sliding window

Abstract

This invention relates to a sliding window-based method and system for measuring flow cardinality. The method includes: acquiring a high-speed network data packet stream; the high-speed network data packet stream includes data packets; generating flow elements based on the data packets, and parsing the data packets to generate flow tags and flow element tags; performing a hash operation on the flow element tags to generate a calculation result; storing the flow elements in a virtual register group based on the calculation result; combining the virtual registers to generate a merged result; and performing flow cardinality estimation on the merged result to generate an estimation result. By introducing virtual register technology, on-chip space resource overhead is reduced, and the problem of not being able to simultaneously measure the cardinality of multiple flows is solved. By eliminating noise by separately estimating the cardinality of the first and second parts of the virtual registers and then subtracting them, the number of register accesses for each cardinality estimation is reduced, thereby improving the efficiency of cardinality lookup.

Description

Technical Field

[0001] This invention relates to the field of network measurement technology, and in particular to a method and system for measuring flow cardinality based on a sliding window. Background Technology

[0002] In recent years, accurate traffic estimation has become a crucial foundation for applications such as attack detection and load balancing in high-speed networks. With the development of the big data era, network scale and traffic speed are constantly growing and expanding. According to statistics provided by Ericsson, monthly mobile network traffic reached 108 EB in the third quarter of 2022, nearly double the 55 EB in the same period of 2020. In high-speed network environments, traffic measurement methods require substantial high-speed storage resources, which are scarce and expensive, posing a significant challenge to accurate traffic measurement under limited high-speed processing resources. Therefore, it is necessary to improve traffic measurement methods in high-speed networks.

[0003] Currently, there are various flow measurement models in the industry, and we focus on flow cardinality measurement under the sliding time window model. The main challenge of flow cardinality measurement under the sliding time window model lies in how to dynamically update the stored flow information while ensuring measurement accuracy. Existing flow cardinality measurement algorithms are not suitable for the sliding time window model because the insertion operation of elements in the sketch is randomized, and the algorithms lose the temporal information of flow elements in order to save on-chip space consumption.

[0004] To address the flow cardinality measurement algorithm problem under the sliding time window model, we aim to reduce the computational and storage overhead of the detection system while ensuring a certain level of accuracy, and simultaneously minimize the system's impact on the network. Key technical challenges include: reducing the space overhead for storing all flow information; ensuring the real-time performance and effectiveness of sampled flow; guaranteeing the integrity of flow element information; and improving the algorithm's accuracy and robustness.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] Therefore, the technical problem this invention aims to solve is to overcome the limitations of existing stream cardinality estimation algorithms based on compact data digests. These algorithms require bit-level or register-level memory sharing to reduce on-chip space overhead, resulting in mixed stream information and requiring consideration of noise from other streams when estimating the cardinality of each stream. Furthermore, each stream cardinality estimation requires scanning the entire sketch, thus hindering real-time querying. Additionally, due to storage space limitations, the temporal information of stream elements cannot be recorded, making it unsuitable for sliding time window models.

[0007] To address the aforementioned technical problems, a first aspect of the present invention provides a method for measuring flow cardinality based on a sliding window, the method comprising:

[0008] Acquire a high-speed network data packet stream; the high-speed network data packet stream includes data packets;

[0009] Stream elements are generated based on the data packets, and stream tags and stream element tags are generated by parsing the data packets.

[0010] The stream element tags are hashed to generate the calculation result;

[0011] The stream element is stored in the virtual register group based on the calculation result;

[0012] The virtual registers are combined and a merged result is generated.

[0013] The merged results are used to estimate the flow basis, generating an estimation result.

[0014] In one embodiment of the present invention, the step of storing the stream element to the virtual register group according to the calculation result further includes:

[0015] If the calculation result is less than a preset threshold, the stream element is stored in the first part of the virtual register;

[0016] If the calculation result is greater than or equal to the preset threshold, the stream element is stored in the second part of the virtual register.

[0017] In one embodiment of the present invention, the stream element is stored in the first part and the second part of the virtual register group according to a preset ratio.

[0018] In one embodiment of the present invention, the step of combining the virtual registers and generating a merged result further includes:

[0019] The merged result is the maximum value in the virtual register.

[0020] In one embodiment of the present invention, the step of estimating the flow cardinality of the merging result to generate the estimation result further includes:

[0021] Perform flow cardinality estimation on the flow elements in the first part to generate the first result;

[0022] Flow cardinality estimation is performed on the flow elements in the second part to generate a second result;

[0023] The estimation result is calculated based on the first result and the second result.

[0024] In one embodiment of the present invention, the flow cardinality of the flow elements in the first part is estimated, and the first result is generated using the following formula:

[0025]

[0026] Where α is the deviation correction constant, s is the number of virtual registers allocated to the data packet stream, and v f The value of the virtual register corresponding to the stream label.

[0027] In one embodiment of the present invention, the flow cardinality of the flow elements in the second part is estimated, and the second result is generated using the following formula:

[0028]

[0029] Where α is the deviation correction constant, s is the number of virtual registers allocated to the data packet stream, and v f The value of the virtual register corresponding to the stream label.

[0030] In one embodiment of the present invention, the estimation result is calculated based on the first result and the second result using the following formula:

[0031]

[0032] in, As the second result, The first result is given.

[0033] In one embodiment of the present invention, the step of calculating the estimation result based on the first result and the second result further includes:

[0034] If the second result is less than the first result, then the estimated result is the second result.

[0035] A second aspect of the present invention provides a flow cardinality measurement system based on a sliding window, the system comprising: an acquisition module, a parsing module, and a calculation module;

[0036] The acquisition module is configured to acquire a high-speed network data packet stream; the high-speed network data packet stream includes data packets.

[0037] The parsing module is configured to: generate stream elements based on the data packet, and parse the data packet to generate stream tags and stream element tags;

[0038] The calculation module is configured to: perform a hash operation on the stream element labels to generate a calculation result; store the stream elements in a virtual register group according to the calculation result; merge the virtual register groups to generate a merge result; and perform stream cardinality estimation on the merge result to generate an estimation result.

[0039] The technical solution of the present invention has the following advantages over the prior art:

[0040] The present invention discloses a sliding window-based method and system for measuring the cardinality of streams. By introducing virtual register technology, it reduces on-chip space resource overhead and solves the problem of the inability to simultaneously measure the cardinality of multiple streams. Noise is eliminated by estimating the cardinality of the first and second parts of the virtual register separately and then subtracting the results, thereby reducing the number of register accesses required for each cardinality estimation and improving the efficiency of cardinality lookup. Attached Figure Description

[0041] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0042] Figure 1 This is a flowchart of a flow cardinality measurement method and system based on a sliding window provided by the present invention;

[0043] Figure 2 This invention provides a method for measuring flow cardinality based on a sliding window and a statistical chart in the system with a sliding time window of 10 seconds and an on-chip space of 2MB.

[0044] Figure 3 This invention provides a method for measuring flow cardinality based on a sliding window and a statistical chart in the system with a sliding time window of 10 seconds and an on-chip space of 4MB.

[0045] Figure 4 This invention provides a method for measuring flow cardinality based on a sliding window and a statistical chart in the system with a sliding time window of 20 seconds and an on-chip space of 2MB.

[0046] Figure 5 This invention provides a method for measuring flow cardinality based on a sliding window and a statistical chart in the system with a sliding time window of 20 seconds and an on-chip space of 4MB.

[0047] Figure 6This invention provides a method for measuring flow cardinality based on a sliding window and a statistical chart in the system with a sliding time window of 60 seconds and an on-chip space of 2MB.

[0048] Figure 7 This invention provides a method for measuring flow cardinality based on a sliding window and a statistical chart of the system with a sliding time window of 60 seconds and an on-chip space of 4MB.

[0049] Figure 8 This is a system architecture diagram of a flow cardinality measurement method and system based on a sliding window provided by the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0051] Reference Figure 1 As shown, in a first aspect, the present invention provides a flow cardinality measurement method based on a sliding window, the method comprising:

[0052] S100, acquire a high-speed network data packet stream; the high-speed network data packet stream includes data packets;

[0053] In practical applications, high-speed network data packet streams are acquired. These streams include data packets. This application maintains a register group M consisting of m physical registers on-chip. Each high-speed network data packet stream is allocated s virtual registers, each corresponding to a physical register. Different high-speed network data packet streams may share the same physical registers. Physical register data within a sliding time window is stored off-chip. Every basic time unit t (set to 1 second in this application), register group M on-chip is downloaded to off-chip. A queue consisting of w register groups is maintained off-chip, where w is the sliding window size.

[0054] S200, Generate a stream element based on the data packet, and parse the data packet to generate a stream label and a stream element label;

[0055] In practical applications, after acquiring multiple data packets, if the target IP address in the data packets is set, the target IP address in each data packet is extracted as a stream element. For each acquired data packet pkt, the data packet pkt is parsed to extract the stream label f ​​and the stream element label e of the data packet pkt.

[0056] S300, perform a hash operation on the stream element label to generate a calculation result;

[0057] In practical applications, the hash operation is performed on the parsed stream element label e to obtain the result h(e), where h(e) is in the range [0, 2]. 32 -1], therefore we can regard h(e) as a 32-bit binary string. We denote the first b bits of h(e) as p and the last (32-b) bits as q, where b = log2s and s is the number of virtual registers allocated to the data packet stream.

[0058] S400, the stream element is stored in the virtual register group according to the calculation result;

[0059] In step S400, the step of storing the stream element to the virtual register group according to the calculation result further includes: if the calculation result is less than a preset threshold, the stream element is stored in the first part of the virtual register; if the calculation result is greater than or equal to the preset threshold, the stream element is stored in the second part of the virtual register. The stream element is stored in the first part and the second part of the virtual register group according to a preset ratio.

[0060] In practical applications, the VHLL algorithm requires scanning the entire register set during each flow cardinality estimation denoising process. This significantly increases the frequency of flow cardinality estimation in the sliding time window model, leading to substantial time overhead when using the VHLL algorithm for denoising. To address this issue, this application employs a novel denoising method. We distribute the flow elements in flow label f ​​in different proportions across the first and second halves of the virtual register set corresponding to that network data packet flow. The first half is the first part, and the second half is the second part. Noise from network data packet flows other than flow label f ​​can be considered uniformly distributed in the virtual register corresponding to flow label f. Therefore, this application uses a method of estimating the flow cardinality of the first and second halves of the virtual register separately and then subtracting the results to eliminate noise. The denoising effect is better when the flow elements are distributed in a preset ratio of 1:5 across the two parts of the virtual register.

[0061] We use the value of the operation result h(e) to determine which part of the virtual register the stream element should be allocated to. The range of the operation result h(e) is [0, 2]. 32 -1], if the value of the calculation result h(e) is less than the preset threshold 2 32 If / 6, then the stream element should be allocated in the first half of the virtual register, that is, the first part of the virtual register, with the index / 6. Otherwise, the stream element should be allocated in the second half of the virtual register, with the index as... The first 1 in q represents the rank value of that element. For example, if q = 0.001..., then rank = 4. Furthermore, to map each virtual register to a specific physical register, we need a hash function H and a random number array R. Assuming the updated virtual register index is `index`, then the index of the physical register to be updated is... The following is abbreviated as

[0062] S500, combine the virtual registers and generate a merged result;

[0063] In step S500, the step of combining the virtual registers and generating a merged result further includes: the merged result is the maximum value in the virtual registers.

[0064] In practical applications, before cardinality estimation, we first perform a merging operation on the w register groups under the chip. Since each register group has m physical registers, each physical register will have w values. We select the maximum value among these w values ​​as the merged value of the physical register. Let the merged value of the register with index i be denoted as M′[i], then M′[i] = max(M0[i], M1[i], ..., M w-1 [i]), where M k [i] is the value of the physical register with index i in the k-th register group, where 0 ≤ k < w. For a given stream label f, we construct a set of virtual register values ​​v corresponding to it. f , Where H is a hash array and R[s] is the value at index S of the random number array.

[0065] S600, perform flow basis estimation on the merged result to generate an estimation result.

[0066] In step S600, the step of estimating the flow cardinality of the merged result to generate an estimation result further includes: estimating the flow cardinality of the flow elements in the first part to generate a first result; estimating the flow cardinality of the flow elements in the second part to generate a second result; and calculating the estimation result based on the first result and the second result. The flow cardinality estimation of the flow elements in the first part to generate the first result uses the following formula:

[0067]

[0068] Where α is the deviation correction constant, s is the number of virtual registers allocated to the data packet stream, and v f The value of the virtual register corresponding to the stream label. Stream cardinality estimation is performed on the stream elements in the second part, and the second result is generated using the following formula:

[0069]

[0070] Where α is the deviation correction constant, s is the number of virtual registers allocated to the data packet stream, and v f The value of the virtual register corresponding to the stream label. The estimation result is calculated based on the first result and the second result using the following formula:

[0071]

[0072] in, As the second result, The first result is taken as the estimated result. The step of calculating the estimated result based on the first result and the second result further includes: if the second result is less than the first result, then the estimated result is the second result.

[0073] In practical applications, cardinality estimation is performed on-chip. We set a timer colck, and every time colck elapses one second, we download the virtual register array maintained on-chip to the off-chip. If the number of physical register groups maintained on-chip exceeds w, the earliest one is discarded. Then, cardinality estimation is performed on the first half and the second half of the virtual register. Among them, the flow cardinality estimation of the flow elements of the first part of the virtual register is as shown in Equation (1), where α is the deviation correction constant, and the commonly used value is α. 16 =0.673, α 32 =0.697, α 64 =0.709, since equation (1) has a large estimation bias for small flows, when the estimation result is less than At that time, use equation (4) to estimate the flow cardinality of the flow elements in the first part of the virtual register.

[0074]

[0075] Where N1 is V f The proportion of 0 in the first part. When estimating the flow cardinality of the flow elements in the second part of the virtual register, as shown in equation (2), α is the deviation correction constant, and the commonly used value is α. 16 =0.673, α 32 =0.697, α 64 =0.709. Since the estimation deviation of the small flow by equation (2) is large, when the estimation result is less than 0.709, the flow cardinality of the flow elements in the second part of the virtual register is estimated by equation (5).

[0076]

[0077] Where N2 is V fThe proportion of 0 in the second part. (This is the result.) and Then, the final estimation result can be calculated using equation (3). If the following occurs... Less than In this case, since the second result of the stream elements in the second part of the virtual register should be much larger than the first result, it is assumed that the vast majority of the stream elements in the first part of the virtual register come from noise from other high-speed network data packet streams, and therefore the result is returned directly. As an estimate.

[0078] Reference Figures 2 to 7 As shown, since there is currently no research on cardinality estimation for each flow under the sliding time window model, this application compares the experimental results of the algorithm under different on-chip storage space and sliding time window sizes. The SVHLL algorithm was tested under the conditions of 2MB and 4MB on-chip space and sliding time window sizes of 10 seconds, 20 seconds, and 60 seconds, respectively. The parameter s = 128 was set in the algorithm. Each point in the figure represents the estimation result of a flow, and the cardinality interval is divided into logarithmic intervals. The horizontal axis represents the actual cardinality of the flow, and the vertical axis represents the estimated cardinality of the flow. Figure 2 This assumes an on-chip storage space of 2MB and a sliding time window size of 10 seconds. Figure 3 This assumes 4MB of on-chip storage and a sliding time window size of 10 seconds. Figure 4 This assumes an on-chip storage space of 2MB and a sliding time window size of 20 seconds. Figure 5 This assumes 4MB of on-chip storage and a sliding time window size of 20 seconds. Figure 6 This assumes 2MB of on-chip storage and a sliding time window size of 60 seconds. Figure 7 For a case where the on-chip space is 4MB and the sliding time window size is 60 seconds, the performance indicators under different experimental conditions are shown in Table 1 and Table 2.

[0079] Table 1:

[0080] ARE 2.9279 4.4537 8.6334 MAE 4.0497 5.8557 10.5816

[0081] Table 2:

[0082] ARE 1.6549 2.4103 4.5682 MAE 2.5571 3.4524 5.9075

[0083] Table 1 shows the performance metrics for different sliding window sizes under a 2MB on-chip space condition, and Table 2 shows the performance metrics for different sliding window sizes under a 4MB on-chip space condition. Where ARE is the mean relative error and MAE is the mean absolute error.

[0084] From a sliding time window of 10 seconds to 60 seconds, it can be seen that the SVHLL algorithm performs well in the estimation of the high-current part. Furthermore, the ARE and MAE are both less than 11 under the condition of 2MB on-chip storage space, and both the ARE and MAE are less than 6 under the condition of 4MB on-chip storage space, with the error within the allowable range.

[0085] Reference Figure 8 As shown, in a second aspect, this application provides a flow cardinality measurement system based on a sliding window, the system comprising: an acquisition module 100, a parsing module 200, and a calculation module 300;

[0086] The acquisition module 100 is configured to acquire a high-speed network data packet stream; the high-speed network data packet stream includes data packets.

[0087] The parsing module 200 is configured to: generate stream elements based on the data packet, and parse the data packet to generate stream tags and stream element tags;

[0088] The calculation module 300 is configured to: perform a hash operation on the stream element labels to generate a calculation result; store the stream elements in a virtual register group according to the calculation result; combine the virtual register groups to generate a merge result; and perform stream cardinality estimation on the merge result to generate an estimation result.

[0089] The effects of applying the aforementioned method in the above system can be found in the description of the aforementioned method embodiments, and will not be repeated here.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for measuring flow cardinality based on a sliding window, characterized in that, The method includes: Acquire a high-speed network data packet stream; the high-speed network data packet stream includes data packets; Stream elements are generated based on the data packets, and stream tags and stream element tags are generated by parsing the data packets. The hash operation is performed on the stream element tags to generate the operation result; The stream element is stored in a virtual register group according to the calculation result; wherein, if the calculation result is less than a preset threshold, the stream element is stored in the first part of the virtual register; if the calculation result is greater than or equal to the preset threshold, the stream element is stored in the second part of the virtual register. The virtual registers are combined and a merged result is generated. The merged results are subjected to flow cardinality estimation to generate an estimation result; wherein, the flow cardinality of the flow elements in the first part is estimated to generate a first result; the flow cardinality of the flow elements in the second part is estimated to generate a second result; if the second result is less than the first result, then the estimation result is the second result; the estimation result is calculated using the following formula: in, As the second result, The first result is given.

2. The flow cardinality measurement method system based on a sliding window according to claim 1, characterized in that: The stream elements are stored in the first and second parts of the virtual register group according to a preset ratio; wherein the preset ratio is 1:

5.

3. The flow cardinality measurement method based on a sliding window according to claim 1, characterized in that, The step of combining the virtual registers and generating the merged result further includes: The merged result is the maximum value in the virtual register.

4. The flow cardinality measurement method based on a sliding window according to claim 1, characterized in that, The flow cardinality of the flow elements in the first part is estimated, and the first result is generated using the following formula: in, This is the deviation correction parameter, where s is the number of virtual registers allocated to the data packet stream. The value of the virtual register corresponding to the stream label.

5. The flow cardinality measurement method based on a sliding window according to claim 1, characterized in that, The flow cardinality of the flow elements in the second part is estimated, and the second result is generated using the following formula: in, This is the deviation correction parameter, where s is the number of virtual registers allocated to the data packet stream. The value of the virtual register corresponding to the stream label.

6. A flow cardinality measurement system based on a sliding window, characterized in that, The system includes: an acquisition module, a parsing module, and a calculation module; The acquisition module is configured to acquire a high-speed network data packet stream; the high-speed network data packet stream includes data packets. The parsing module is configured to: generate stream elements based on the data packet, and parse the data packet to generate stream tags and stream element tags; The calculation module is configured to: perform a hash operation on the stream element labels to generate a calculation result; store the stream elements into a virtual register group according to the calculation result; combine the virtual register groups to generate a merged result; and perform stream cardinality estimation on the merged result to generate an estimation result. Wherein, if the operation result is less than a preset threshold, the stream element is stored in the first part of the virtual register; if the operation result is greater than or equal to the preset threshold, the stream element is stored in the second part of the virtual register; Flow cardinality estimation is performed on the flow elements in the first part to generate a first result; flow cardinality estimation is performed on the flow elements in the second part to generate a second result; if the second result is less than the first result, then the estimation result is the second result; the estimation result is calculated using the following formula: in, As the second result, The first result is given.

Images