A network large flow elasticity measurement method supporting traffic jitter

Abstract

The present invention discloses a network large flow elasticity measurement method supporting traffic jitter, the method comprising: firstly parsing the network packet and extracting a quintuple as a flow identifier; then designing a small flow filter based on a scalable Sketch cycle chain, and then proposing a filter expansion and contraction strategy that can adaptively adjust the number of Sketches in the cycle chain according to the dynamic change of the network packet rate, so as to ensure that all packets within a given time period are recorded, so as to continuously and accurately filter the small flow; then combining a multi-segment scalable hash algorithm, a large flow recorder based on a scalable hash table is designed, which can expand or contract segments according to the dynamic change of the number of recorded network large flows, so as to record all large flows as much as possible, while improving the utilization rate of storage space; according to the solution of the present invention, the problem of insufficient query accuracy and the problem of being unable to realize adaptive measurement in large flow measurement when the traffic surges on a high-speed network link is effectively solved, and the accuracy and range of large flow measurement are further improved.

Description

Technical Field

[0001] The present invention relates to the field of network measurement, and more specifically, to a network large flow elasticity measurement method supporting flow jitter. Background Art

[0002] Network measurement is to analyze the network status and behavior characteristics and quantify various network indicators. It plays an important role in network management, network operation and maintenance, network security, network optimization and traffic engineering. Large flow identification and extraction play an important role in network measurement. However, since network measurement is limited by traditional network architecture and terminals, the traffic compression methods in the measurement methods focus on sampling technology and data flow technology. Sampling technology refers to the representative group subset extracted from the overall network flow to perform traffic statistics, and infer the state of the original network traffic through the representative group subset. However, due to the randomness and uncertainty of sampling, it will cause high errors, so the measurement results using sampling technology are not accurate. At present, data flow technology with Sketch as the measurement carrier is a relatively mature technology and has been widely used in large flow measurement in network measurement.

[0003] Sketch is a data structure designed to store huge amounts of information data in a small amount of storage space. First, the network traffic data is modeled into key values ​​(k, v), and then the massive information is hashed to the corresponding positions through the calculation of the hash function. Finally, the values ​​at each position are counted and used as the estimated approximation of the overall data. In order to better adapt to large-scale network measurements, Sketch sacrifices a certain degree of accuracy in exchange for lower storage space consumption and fast calculation processing time to ensure the real-time nature of network measurements. Minimum Count Sketch (Count-Min Sketch, CM Sketch) is a typical Sketch, such as Figure 1 As shown, CM Sketch consists of d arrays, each of which has a corresponding hash function h i (1≤i≤d) and w counters. When inserting a data item, first model it as a key value (k,v) and use the corresponding hash function h i (k)(1≤i≤d) maps the data item to the corresponding counter in each array, and adds v to the value of each counter. When querying a data item, each array needs to be queried separately to obtain the corresponding counter value in each array after being mapped by the hash function, and then the minimum value among these values ​​is taken as the measurement result of the data item.

[0004] At present, Cold Filter is a more efficient network large flow measurement solution based on Sketch method, which uses a two-layer architecture of filter idea to filter small flows and count large flows. [Solution 1] Figure 2As shown in FIG. 1 , the data structure of CF is composed of w1 counters and w2 counters, and both are associated with d1 hash functions h(.) and d2 hash functions g(.) to form L1 layer and L2 layer. The counter sizes of L1 layer and L2 layer are δ1 and δ2 bits respectively, where L1 layer is used to record most of the data flow to filter small flows. When the count value exceeds the range that L1 layer can count, it is judged as a large flow and counted to L2 layer to process its data information.

[0005] The scheme is divided into update operation and query operation: the update operation first finds the corresponding position in L1 through hash function calculation. If the minimum value is less than the range of L1 count, the minimum value is directly updated at this time, such as Figure 2 The counting range of L1 is 15, and the minimum value corresponding to the data item e mapped by the function h(.) is 13, then it is only necessary to increase the counter value by 1. If there are multiple minimum values, then all minimum values ​​are updated, such as Figure 2 If the minimum counter value in L1 is 15, then all counters with a value of 15 are incremented by 1. At this time, L1 has exceeded the counting range, and the same operation is performed in L2. The query operation also first finds the corresponding position of L1 through the hash function. If the minimum value is less than the L1 counting range, report the value. Conversely, search L2 in the same way. If the count of L2 does not exceed the counting range, report the sum of L1 and L2. If both L1 and L2 exceed the counting range, report that they exceed the specified threshold, for example Figure 2 If all minimum value counters in L1 reach the maximum count range of 15, a query operation is performed in L2. At this time, the counter values ​​in L2 also reach the maximum count range of 15, and it is reported as a large flow.

[0006] When measuring networks under large-scale traffic, as the number of large flows increases significantly, the network measurement solution of the traditional filter architecture cannot flexibly change according to the changes in traffic. Elastic Sketch is a more efficient elastic network measurement solution that can flexibly measure large flows in the network. [Solution 2] Figure 3 As shown in Figure 1, its architecture consists of two parts: the heavy part and the light part. The heavy part uses a hash array to record the relevant information of the flow. Each hash bucket contains: flow ID (key), positive vote (vote + ), flag, vote - ). The flow ID is a unique identifier representing the network flow extracted from the five-tuple of the packet; vote + Refers to the number of packets belonging to the flow, that is, the flow size; vote - Refers to the number of other packets mapped to the same hash bucket; flag indicates whether the small flow part contains this flow. The light part uses CM Sketch to count the small flows kicked out from the heavy part.

[0007] Most flows in network traffic are small, so the memory required for the light part is usually much larger than that for the heavy part. In order to adapt to the available bandwidth, the light part can be compressed. The core idea is to group counters and then compress the counters in the same group into one counter through a merging strategy. Figure 4 As shown in A d The counters in the array are divided into z groups, A d 1 [1] A d 2 [1]...A d z The corresponding counters (51, 9, 99) of [1] are compressed into B through the maximum compression merging strategy. d Counter (99) in [1].

[0008] Since the number of large flows in network traffic varies, the size of the heavy part cannot be determined. This means that when the number of hash buckets in the heavy part exceeds the counting range, the large flow information cannot be recorded. In order to adapt to the traffic size distribution, the heavy part can be expanded. Figure 5 As shown, the original number of buckets of the heavy part is 4, and the number of buckets becomes 8 after the expansion operation. f2 is mapped to the position of f3 through the hash function h(f2)%8, and f3 is then mapped to the copy part by calculating the hash function h(f3)%8=6. At this time, f3 is replaced by f2.

[0009] However, the above existing network large flow measurement solutions have the following defects:

[0010] [Solution 1] A two-layer architecture based on the filter concept is designed. Network traffic is counted through different hash functions. The lower layer filters small flows by setting thresholds, and the higher layer records the filtered large flows. The size of the counting units of the two layers can be flexibly set according to the measurement task, which improves the space utilization of Sketch. However, when the network traffic surges, too many small flows will generate a large number of hash conflicts, the counting units of the lower layer will quickly reach the threshold, and the higher layer will immediately report overflow, which will cause the filter to fail and the accuracy to be low.

[0011] [Solution 2] A heavy-light dual-part measurement architecture was designed. The heavy part uses a hash array to record the traffic information of each flow in the network, and uses a voting strategy to select large flows with more positive votes, while the expelled small flows are stored in the light part composed of CM Sketch. However, due to the limited capacity of the hash array of the heavy part, the large number of small flows causes a large number of hash conflicts in the traffic statistics. Multiple small flows will be mapped to the same bucket of the large flow, resulting in too many negative votes in the bucket, which will cause the large flow to be kicked out, which has a certain impact on the accuracy of large flow measurement. Summary of the invention

[0012] In view of the above problems in the prior art, the present invention proposes a network large flow elasticity measurement method that supports traffic jitter based on the idea of ​​filtering small flows first and then recording large flows. First, a scalable cyclic filter is designed, and then a filter expansion and contraction strategy that can adapt to the dynamic changes in the network packet rate is proposed to ensure that all packets within a given time period are recorded, so as to continuously and accurately filter small flows; then a large flow recorder based on a multi-segment scalable hash algorithm is designed, which can expand or contract segments according to the dynamic changes in the number of recorded network large flows, so as to record all large flows as much as possible, while improving the utilization of storage space.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is:

[0014] The present invention proposes a network large flow elasticity measurement method supporting flow jitter, the steps are as follows:

[0015] Step 1: Network traffic analysis: When a packet is received, first parse its protocol header, extract the five-tuple field, and calculate the flow identifier fid;

[0016] Step 2: Small flow filtering: The flow identifier of the network packet parsed in step 1 is used as the input of the small flow filter to preliminarily analyze the size of the flow to which the packet belongs; through the insertion operation, the statistical Sketch update operation and the Sketch loop chain expansion operation, the small flow is continuously filtered. The Sketch is a data structure designed to store huge information data with a small storage space; if it is determined to be a large flow, the large flow recording process is entered;

[0017] Step 3: Recording large flows: The network traffic data determined as large flows in the small flow filtering process in step 2 is input into the large flow recorder, the storage location is calculated through a multi-segment scalable hash algorithm, and the large flows are counted through insertion and scaling operations;

[0018] Step 4: Identify the large flow: According to the measurement results obtained in step 3, identify the large flow.

[0019] Further, in the step 1, the five-tuple field of the network packet includes the source IP address, the destination IP address, the source port, the destination port and the IP protocol type;

[0020] Furthermore, in the step 2:

[0021] The low-flow filter is composed of a scalable circular chain consisting of multiple Sketches and a statistical Sketch; each Sketch consists of d rows and w columns of counters, and each counter records the number of packets mapped to the position; each Sketch has a start time and an end time, which serves as a basis for judging whether the circular chain is extended or contracted when the currently counted Sketch is full; the Sketch circular chain completely records all packets within a fixed time period, and the statistical Sketch records the weighted average of the counters at the same position of all Sketches in the circular chain as a basis for judging whether the incoming packet belongs to a low-flow; in order to prevent the performance of the filtering function from being degraded due to the gradual overflow of a large number of counters, when the number of counters reaching the threshold in the current counting Sketch of the circular chain exceeds the preset ratio, the number of Sketches is adaptively increased or deleted according to whether the Sketch in the circular chain can completely record all packets in a time period, so as to ensure that the low-flow filter always maintains a high accuracy rate when the network packet rate fluctuates;

[0022] Insert operation: When the small flow filter receives the flow identifier fid of a packet, it first maps a counter in each bucket of the current Sketch through d independent hash functions, and then finds the minimum counter among all mapped counters. If the minimum counter value is less than the specified threshold, it is increased by one, otherwise it directly enters the statistical Sketch to determine whether it is a large flow; when multiple packets are hashed to the same position, the value of the minimum counter is accumulated; after the counter value increases, if the number of counters that reach the threshold in the current Sketch exceeds a certain proportion, the statistical Sketch update operation and the circular chain expansion operation are successively executed; next, a counter is mapped in each bucket of the statistical Sketch, and the minimum counter is found; if the minimum counter does not reach the threshold, it is determined to be a small flow, otherwise it is determined to be a large flow and enters the large flow recording process;

[0023] Statistical Sketch update operation: If the number of counters that reach the threshold in the current Sketch exceeds the preset ratio, the Sketches in the loop chain are assigned different weights in chronological order, and then the weighted average of the counters at the same position in each Sketch is added to the corresponding counter of the statistical Sketch;

[0024] Sketch loop chain scaling operation: After the update operation is completed, first obtain the number of Sketches n in the loop chain, the start time t0 of the first Sketch, and the current time t; if the time length from t to t0 is less than the given time period T, then add a new Sketch and add it to the end of the loop chain as a new counting Sketch; otherwise, start from the first Sketch and get the end time t of each Sketch in turn i, where 1≤i≤n-1, and judge the distance t i Is the time length less than T? If not, delete the Sketch; repeat until the distance t is less than t. i Until the time length is less than T; finally, reset the i-th Sketch to continue counting.

[0025] Furthermore, in step three:

[0026] The large flow recorder is a scalable hash table composed of multiple logical segments with a variable number; each segment is composed of a fixed number of large flow items and is indexed by a corresponding sub-hash function; each large flow item contains a corresponding flow fingerprint fp and a counter that records the number of packets, where the flow fingerprint fp is generated by the flow identifier fid through a hash function; when the proportion of large flows that reach the packet number threshold in the large flow recorder is too large, new segments will be added; when the proportion of large flows that reach the packet number threshold in the large flow recorder is too small, segments will be reduced to adapt to the jitter of network traffic; when a packet belonging to a new flow arrives, if all index positions are full, the large flow item with the smallest number of packets is found, and then the counter is decayed with a certain probability to prevent small flows from being misjudged as large flows;

[0027] Sub-hash function: Assuming the segment length is l, the sub-hash function first randomly selects m different bits from the flow fingerprint fp, where m ≥ log2l, and randomly arranges them into an m-bit bit string, and then takes the modulus of l;

[0028] Insertion operation: When a packet passes through the small flow filter, the flow fingerprint fp is first calculated according to its flow identifier fid; then an index position is mapped in each segment through the sub-hash function, and the corresponding n large flow items are matched with fp; if a large flow item is successfully matched, the counter in the large flow item is increased by 1; if the counter reaches a certain threshold, it is determined to be a large flow; at this time, if the proportion of large flows in the large flow recorder exceeds the preset upper limit, a new segment is added to accommodate the subsequent large flows; if the match fails, but the index position of a segment is empty, the flow is inserted into it; if the index positions of all segments are full, the smallest large flow item is found and the counter is increased with probability b -C Minus 1; where C is the counter value and b is a constant slightly greater than 1; if the counter value becomes 0, a new stream is inserted and the counter is set to 1; after the counter decays, if the proportion of large streams in the large stream recorder is lower than the preset lower limit, a segment is deleted and the large stream recorded in the segment is moved to the corresponding index position of the remaining segments;

[0029] Scaling operation: If the proportion of large flows in the large flow recorder exceeds the preset upper limit, a new empty segment is added and a new sub-hash function algorithm is generated to index the segment; if the proportion of large flows in the large flow recorder is lower than the preset lower limit, the number of segments n in the large flow record table is first obtained, and the segment with the least number of large flows stored in all segments is found as the segment to be deleted; then, starting from the first flow stored in the segment, each flow in the segment is indexed in turn by its flow fingerprint fp in the remaining segments; if there is an empty index position, the flow is inserted into it; if there is no empty index position, the minimum counter value in all index positions is found and compared with the number of packets recorded in the flow; if the minimum counter value in the index position is less than the number of packets recorded in the flow, the flow is replaced with the index position corresponding to the minimum counter, otherwise it is directly discarded; after all flows in the segment are processed, the segment is deleted.

[0030] Furthermore, in step 4, when performing large flow identification, first obtain the number of segments n and the segment length l of the large flow recorder; then check the counter in the large flow item one by one in each segment; if the counter reaches the threshold, output the corresponding flow fingerprint and flow size, wherein the flow size refers to the sum of the counter value and the small flow filtering threshold.

[0031] The characteristics of this method that distinguish it from existing methods are:

[0032] (1) The small flow filter ensures the continued effectiveness of filtering small flows by dynamically adjusting the length of the Sketch loop chain to adapt to the fluctuations in the network packet rate; (2) The large flow recorder ensures the accurate recording of all large flows by dynamically expanding and contracting segments to adapt to the dynamic changes in the number of large flows in the network; (3) The present invention accurately records all large flows on the basis of ensuring continuous filtering of small flows, realizes accurate measurement of large flows under network traffic jitter, and ensures high utilization of storage space.

[0033] A network large flow elasticity measurement method supporting flow jitter according to an embodiment of the present invention has at least the following technical effects:

[0034] 1. In view of the problem that existing large flow measurement schemes are prone to misjudging small flows as large flows when the network packet rate surges, this patent proposes a small flow adaptive filtering method that perceives the network packet rate, and designs a small flow filter based on a scalable Sketch cycle chain. The number of Sketches is adaptively increased or decreased according to the dynamic changes in the network packet rate, so that the Sketch cycle chain always fully records all packets within a given time period, so that small flows can still be accurately filtered when the packet rate fluctuates violently. When the packet rate surges, the Sketch cycle chain adaptively adds a Sketch to avoid overloading the existing Sketch and causing its filtering function to fail, thereby ensuring the continued effectiveness of the small flow filter; when the packet rate drops sharply, the Sketch cycle chain adaptively deletes the Sketch, while ensuring the effectiveness of the filtering function, reducing the storage overhead of the small flow filter.

[0035] 2. In view of the problem that the existing large flow measurement scheme cannot record all large flows due to fixed capacity when the network traffic jitters, the present invention proposes a large flow elastic recording method that supports network traffic jitter, designs a large flow recorder based on a scalable hash table, and adaptively adjusts the storage capacity according to the dynamic changes in the number of large flows in the network, so that the large flow recorder always records all large flows as much as possible, thereby ensuring the accuracy of large flow records when the network traffic jitters. When the number of large flows in the network traffic surges, the large flow recorder adds a new segment to accommodate the newly emerged large flows; when the number of large flows in the network traffic drops sharply, the large flow recorder reduces a segment, while ensuring the accuracy of large flow records, while improving the space utilization of the large flow recorder. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the CM Sketch data structure provided by the prior art of the method of the present invention.

[0038] Figure 2 It is a schematic diagram of the Cold Filter data structure and algorithm provided by the prior art of the method of the present invention.

[0039] Figure 3 It is a schematic diagram of the elastic Sketch data structure provided by the prior art of the method of the present invention.

[0040] Figure 4It is a schematic diagram of the elastic Sketch light part compression algorithm provided by the prior art of the method of the present invention.

[0041] Figure 5 It is a schematic diagram of the elastic Sketch heavy part expansion algorithm provided by the prior art of the method of the present invention.

[0042] Figure 6 It is a schematic diagram of the network mass flow elasticity measurement solution architecture provided by the method of the present invention.

[0043] Figure 7 It is a schematic diagram of the data structure of the small flow filter in the method of the present invention.

[0044] Figure 8 It is a schematic diagram of the data structure of the large stream recorder in the method of the present invention.

[0045] Fig. 9 It is an overall flow chart of large flow measurement in an embodiment of the present invention.

[0046] Fig.10 It is a flow chart of inserting a small flow filter in an embodiment of the present invention.

[0047] Fig.11 This is a flowchart of updating the statistical Sketch in an embodiment of the present invention.

[0048] Fig.12 It is a flowchart of Sketch circular chain expansion and contraction in an embodiment of the present invention.

[0049] Fig.13 This is the high-volume recorder insertion process in the embodiment of the present invention.

[0050] Fig.14 It is a flow chart of segmented scaling of a large stream recorder in an embodiment of the present invention.

[0051] Fig.15 It is a flow chart of large flow identification in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Architecture Example:

[0054] Reference Figures 6 to 8 The embodiment of the present invention provides a network large flow elasticity measurement architecture supporting flow jitter, including:

[0055] The small flow filter filters the small flow by setting the filtering threshold, thereby distinguishing the large flow and the small flow in the network traffic, and then realizing the accurate statistics of the large flow in the network; the small flow filter is composed of a scalable circular chain composed of multiple Sketches and a statistical Sketch, and when the currently counted Sketch in the circular chain is full, the number of Sketches will be adaptively increased or deleted according to whether the Sketch in the circular chain can completely record all the packets in a time period, so as to ensure that the small flow filter always maintains a high accuracy rate when the network packet rate fluctuates;

[0056] like Figure 7 As shown in the figure, each Sketch consists of d rows and w columns of counters, each counter records the number of packets mapped to the position; each Sketch has a start time and an end time, which is used as a basis for judging whether the loop chain should be extended or contracted when the currently counted Sketch is full; the Sketch loop chain completely records all packets within a fixed time period, and the weighted average of the counters at the same position of all Sketches in the Sketch loop chain is counted as a basis for judging whether the incoming packet belongs to a small flow;

[0057] A large flow recorder is used to accurately record large flows in network traffic and accurately count the number of packets in the large flow; the large flow recorder is a scalable hash table composed of a variable number of multiple logical segments, each segment consists of a fixed number of large flow items and is indexed by a corresponding sub-hash function; when the proportion of large flows that reach the packet number threshold in the large flow recorder is too large or too small, new or reduced segments are added to adapt to the jitter of network traffic and improve storage space utilization; when a packet belonging to a new flow arrives, if all index positions are full, the large flow item with the smallest number of packets is found, and then the counter is decayed with a certain probability to prevent small flows from being misjudged as large flows;

[0058] like Figure 8 As shown, the large flow recorder consists of a variable number of logical segments, each segment contains l large flow items, each large flow item contains a corresponding flow fingerprint fp and a counter for recording the number of packets, where the flow fingerprint fp is generated by the flow identifier fid through a hash function; when the data packet P fid When it arrives, we calculate the flow fingerprint fp through the hash function H(.). The sub-hash function first randomly selects different m (m ≥ log2l) bits from the flow fingerprint fp and randomly arranges them into an m-bit bit string, and then takes the modulus of l to generate different segment index positions.

[0059] Method Example:

[0060] Reference Figures 9 to 15 The embodiment of the present invention provides a network large flow elasticity measurement method supporting flow jitter, comprising the following steps:

[0061] like Fig. 9 The figure shows the overall process of large flow measurement of the present invention. When a packet is received, first parse its protocol header, extract the five-tuple field, and calculate the flow identifier fid; then enter the small flow filtering process to determine whether the flow is a small flow; if it is a small flow, directly discard the packet; otherwise enter the large flow recording process to record the flow fingerprints and packet numbers of all large flows as much as possible; after processing the received network packets, enter the large flow identification process to achieve fast and accurate identification and reporting of large flows.

[0062] 1. Network traffic analysis

[0063] When a packet is received, its protocol header is first parsed to extract the five-tuple fields, namely, the source IP address, source port, destination IP address, destination port and IP protocol type, so as to calculate the flow identifier; in this embodiment, fid is used to represent the flow identifier;

[0064] 2. Small flow filtration process

[0065] First, initialize the small flow filter, and then use the parsed network packet flow identifier as the input of the small flow filter to preliminarily analyze the size of the flow to which the packet belongs; through the insertion operation, the statistical Sketch update operation and the loop chain expansion operation, the small flow is continuously and efficiently filtered; if it is determined to be a large flow, it enters the large flow recording process;

[0066] In this step, referring to the above architecture embodiment, the specific process is as follows:

[0067] (1) Low-flow filter insertion:

[0068] like Fig.10 As shown, when the small flow filter receives the flow identifier fid of a packet, it first maps a counter in each bucket of the current Sketch through d independent hash functions, and then finds the minimum counter among all the mapped counters. If the minimum counter value is less than the specified threshold, it is increased by one, otherwise it directly enters the statistical Sketch to determine whether it is a large flow; when multiple packets are hashed to the same position, the value of the minimum counter is accumulated; after the counter value increases, if the number of counters that reach the threshold in the current Sketch exceeds a certain proportion, the statistical Sktech update operation and the circular chain expansion operation are successively executed; next, a counter is mapped in each bucket of the statistical Sketch, and the smallest counter is found; if the minimum counter does not reach the threshold, it is determined to be a small flow, otherwise it is determined to be a large flow and enters the large flow recording process.

[0069] (2) Statistics Sketch updates:

[0070] like Fig.11As shown in the figure, if the current Sketch is full, the statistical Sketch update process is entered; first, the number of rows d, columns w, and number n of Sketches in the circular chain are obtained; then the kth Sketch in the circular chain is assigned a weight of 2 k-1 / (2 n -1), and then perform weighted average on the counters at the same position in each Sketch to obtain the new value of the corresponding counter in the statistical Sketch.

[0071] (3) Sketch cycle chain expansion:

[0072] like Fig.12 As shown, first get the number of Sketches n in the loop chain, the start time t0 of the first Sketch, and the current time t; if the time length from t to t0 is less than the given time period T, then add a new Sketch and add it to the end of the loop chain as a new counting Sketch; otherwise, get the end time t of the i-th (1≤i≤n-1) Sketch in turn i , and determine the distance t i Is the time length less than T? If not, delete the Sketch; repeat until the distance t is less than t. i Until the time length is less than T; finally, reset the i-th Sketch to continue counting.

[0073] 3. High-volume recording process

[0074] The network traffic data that is determined as a large flow in the small flow filtering process is input into the large flow recorder, and the storage location is calculated through a multi-segment scalable hash algorithm. The large flow is accurately and completely counted through insertion and scaling operations.

[0075] In this step, referring to the above architecture embodiment, the specific process is as follows:

[0076] (1) High-speed stream recorder insertion:

[0077] like Fig.13 As shown, when a packet is determined by the small flow filter to belong to a large flow (the flow identifier is fid), it enters the large flow recorder; first, the flow fingerprint is generated by the hash function H(.); then a large flow item is indexed in each segment by the corresponding sub-hash function, and then matched by the flow fingerprint; if the match is successful, the counter of the matching large flow item is increased by 1; at this time, if the total large flow ratio in the large flow recorder is too high, an empty segment is added to record the large flow that arrives subsequently; if the match fails, check whether there is an empty index position; if so, insert the new flow into it and set the counter to 1; otherwise, find the minimum counter among all index positions and add it with probability b. -CDecrease it by 1; where C is the counter value and b is a constant slightly greater than 1; when the counter is reduced to 0, insert the new flow into it and set the counter to 1; after the counter decays, if the total large flow ratio in the large flow recorder is too low, delete a segment to reduce storage space overhead.

[0078] (2) Segmented scaling of large stream recorders:

[0079] like Fig.14 As shown, if the proportion of large flows in the large flow recorder exceeds the preset upper limit, a new empty segment is added, and a new sub-hash function algorithm is generated to index the segment; if the proportion of large flows in the large flow recorder is lower than the preset lower limit, the number of segments n in the large flow record table is first obtained, and the segment with the least number of large flows stored in all segments is found as the segment to be deleted; then, starting from the first flow stored in the segment, each flow in the segment is indexed in turn by its flow fingerprint fp in the remaining segments; if there is an empty index position, the flow is inserted into it; if there is no empty index position, the minimum counter value in all index positions is found and compared with the number of packets recorded in the flow; if it is smaller, the flow is replaced with the index position corresponding to the minimum counter, otherwise it is directly discarded; after all flows in the segment are processed, the segment is deleted.

[0080] 4. Mass flow identification process

[0081] After the received network packets are processed, the large flow can be identified based on the measurement results recorded by the large flow recorder and the set large flow threshold.

[0082] like Fig.15 As shown, first obtain the number of segments n and the segment length l of the large flow recorder. Then check the counter in the large flow item one by one in each segment. If the counter reaches the threshold, output the corresponding flow fingerprint and flow size (i.e., the sum of the counter value and the small flow filtering threshold).

[0083] It should be noted that the present method embodiment and the above-mentioned architecture embodiment are based on the same inventive concept.

[0084] The specific embodiments described herein are merely examples of the spirit of the present invention. A person skilled in the art of the present invention may make various modifications or additions to the specific embodiments described or replace them in a similar manner, but this will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A network flow elasticity measurement method supporting flow jitter, the steps are as follows: Step 1: Network traffic analysis: When a packet is received, first parse its protocol header, extract the five-tuple field, and calculate the flow identifier fid; Step 2, filtering small flows: the flow identifier of the network packet parsed in step 1 is used as the input of the small flow filter to preliminarily analyze the size of the flow to which the packet belongs; the small flow filter is composed of a scalable circular chain composed of multiple Sketches and a statistical Sketch; the Sketch is a data structure designed to store huge information data with a small amount of storage space; each Sketch is composed of d rows and w columns of counters, and each counter records the number of packets mapped to the position; each Sketch has a start time and an end time; the Sketch circular chain completely records all packets within a fixed time period, and the statistical Sketch records the weighted average of the counters at the same position of all Sketches in the circular chain; when the number of counters reaching the threshold in the current count Sketch of the circular chain exceeds the preset ratio, the number of Sketches is adaptively increased or deleted according to whether the Sketch in the circular chain can completely record all packets in a time period; through the insertion operation, the statistical Sketch update operation and the Sketch circular chain expansion and contraction operation, the continuous filtering of small flows is realized, and if it is determined to be a large flow, the large flow recording process is entered; Step 3, large flow record: input the network traffic data determined as large flow in the small flow filtering process of step 2 into the large flow recorder, calculate the storage location through a multi-segment scalable hash algorithm, and count the large flow through insertion and expansion operations; the large flow recorder is a scalable hash table composed of a plurality of logical segments with a variable number; each segment is composed of a fixed number of large flow items and is indexed by a corresponding sub-hash function; each large flow item contains a corresponding flow fingerprint fp and a counter recording the number of packets, wherein the flow fingerprint fp is generated by a flow identifier fid through a hash function; when the proportion of large flows reaching the packet number threshold in the large flow recorder is too large, a new segment will be added, and when the proportion of large flows reaching the packet number threshold in the large flow recorder is too small, the segment will be reduced; when a packet belonging to a new flow arrives, if all index positions are full, find the large flow item with the smallest number of packets, and then decay the counter with a certain probability; decaying the counter with a certain probability means that the counter is caused to decay with a probability b -C Subtract 1, where C is the counter value and b is a constant slightly greater than 1; Step 4: Identify the large flow: According to the measurement results obtained in step 3, identify the large flow.

2. According to claim 1, a network large flow elasticity measurement method supporting flow jitter is characterized in that: The step 2 comprises the following steps: Insert operation: When the small flow filter receives the flow identifier fid of a packet, it first maps a counter in each bucket of the current Sketch through d independent hash functions, and then finds the minimum counter among all mapped counters. If the minimum counter value is less than the specified threshold, it is increased by one, otherwise it directly enters the statistical Sketch to determine whether it is a large flow; when multiple packets are hashed to the same position, the value of the minimum counter is accumulated; after the counter value increases, if the number of counters that reach the threshold in the current Sketch exceeds a certain proportion, the statistical Sketch update operation and the circular chain expansion operation are successively executed; next, a counter is mapped in each bucket of the statistical Sketch, and the minimum counter is found; if the minimum counter does not reach the threshold, it is determined to be a small flow, otherwise it is determined to be a large flow and enters the large flow recording process; Statistical Sketch update operation: If the number of counters that reach the threshold in the current Sketch exceeds the preset ratio, the Sketches in the loop chain are assigned different weights in chronological order, and then the weighted average of the counters at the same position in each Sketch is added to the corresponding counter of the statistical Sketch; Sketch loop chain scaling operation: After the update operation is completed, first obtain the number of Sketches n in the loop chain, the start time t0 of the first Sketch, and the current time t; if the time length from t to t0 is less than the given time period T, then add a new Sketch and add it to the end of the loop chain as a new counting Sketch; otherwise, start from the first Sketch and get the end time t of each Sketch in turn i , where 1≤i≤n-1, and judge the distance t i Is the time length less than T? If not, delete the Sketch; repeat until the distance t is less than t. i Until the time length is less than T; finally, reset the i-th Sketch to continue counting.

3. According to a method for measuring network mass flow elasticity supporting flow jitter according to claim 1, it is characterized in that: The step three comprises the following steps: Sub-hash function: Assuming the segment length is l, the sub-hash function first randomly selects m different bits from the flow fingerprint fp, where m ≥ log2l, and randomly arranges them into an m-bit bit string, and then takes the modulus of l; Insertion operation: When a packet passes through the small flow filter, the flow fingerprint fp is first calculated according to its flow identifier fid; then an index position is mapped in each segment through the sub-hash function, and the corresponding n large flow items are matched with fp; if a large flow item is successfully matched, the counter in the large flow item is increased by 1; if the counter reaches a certain threshold, it is determined to be a large flow; at this time, if the proportion of large flows in the large flow recorder exceeds the preset upper limit, a new segment is added to accommodate the subsequent large flows; if the match fails, but the index position of a segment is empty, the flow is inserted into it; if the index positions of all segments are full, the smallest large flow item is found and the counter is increased with probability b -C Minus 1; where C is the counter value and b is a constant slightly greater than 1; if the counter value becomes 0, a new stream is inserted and the counter is set to 1; after the counter decays, if the proportion of large streams in the large stream recorder is lower than the preset lower limit, a segment is deleted and the large stream recorded in the segment is moved to the corresponding index position of the remaining segments; Scaling operation: If the proportion of large flows in the large flow recorder exceeds the preset upper limit, a new empty segment is added and a new sub-hash function algorithm is generated to index the segment; if the proportion of large flows in the large flow recorder is lower than the preset lower limit, the number of segments n in the large flow record table is first obtained, and the segment with the least number of large flows stored in all segments is found as the segment to be deleted; then, starting from the first flow stored in the segment, each flow in the segment is indexed in turn by its flow fingerprint fp in the remaining segments; if there is an empty index position, the flow is inserted into it; if there is no empty index position, the minimum counter value in all index positions is found and compared with the number of packets recorded in the flow; if the minimum counter value in the index position is less than the number of packets recorded in the flow, the flow is replaced with the index position corresponding to the minimum counter, otherwise it is directly discarded; after all flows in the segment are processed, the segment is deleted.

4. According to claim 1, a network large flow elasticity measurement method supporting flow jitter is characterized in that: In the step 4: When identifying large flows, first obtain the number of segments n and the segment length l of the large flow recorder; then check the counter in the large flow item one by one in each segment; if the counter reaches the threshold, output the corresponding flow fingerprint and flow size, where the flow size refers to the sum of the counter value and the small flow filtering threshold.

5. A network large flow elasticity measurement method supporting flow jitter according to claim 2, characterized in that: The small flow filter based on the scalable Sketch loop chain adaptively increases or decreases the number of Sketches in the loop chain according to the dynamic changes of the network packet rate; when the packet rate surges, the Sketch loop chain adaptively adds Sketches; when the packet rate drops sharply, the Sketch loop chain adaptively deletes Sketches.

6. A network large flow elasticity measurement method supporting flow jitter according to claim 3, characterized in that: The large flow recorder based on scalable hash tables adaptively adjusts the storage capacity according to the dynamic changes in the number of large flows in the network; when the number of large flows in the network traffic increases sharply, the large flow recorder adds a new segment to accommodate the newly emerging large flows; when the number of large flows in the network traffic decreases sharply, the large flow recorder reduces a segment.

Images