A method for identifying network reflow
By designing a sketch structure of multiple one-dimensional hash tables in a programmable switch, the problems of high hardware storage overhead and low recognition accuracy in reflow identification in data center networks are solved, achieving efficient and accurate reflow identification and reducing the frequent small flow error replacement rate.
Patent Information
- Application Number
- CN202410132779.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-31
AI Technical Summary
Existing technologies for reflow identification in data center networks suffer from problems such as high hardware storage overhead, high processing latency, and low identification accuracy when deployed on programmable switches. In particular, sketch-based methods are prone to frequent small flow error replacements in hardware implementation.
In a programmable switch, a sketch structure of multiple one-dimensional hash tables is designed. Each hash table has w storage units. The index value of the data packet is calculated by a hash function, and the traffic size record is replaced according to the probability. A threshold for the number of reflow hits is set to identify reflows, and the table entries are updated through the control plane node to improve the identification accuracy.
It reduces the incidence of frequent small-flow erroneous replacements, improves the accuracy and efficiency of reflow identification, and reduces the erroneous replacement rate by 90% compared to the USS and cocoSketch methods of the same storage size.
Smart Images

Figure CN117714349B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a network recurrent flow identification method, and more particularly to a method for identifying recurrent flows in a data center network through passive measurement. Background Technology
[0002] Data center network traffic exhibits a skewed pattern, meaning that a small percentage of recurrent flows carry the vast majority of data. Quickly identifying these recurrent flows is crucial for data center network monitoring and optimization. In data center network measurement, identifying recurrent flows primarily involves recording their traffic volume and ID (typically composed of an IP 5-tuple or a source / destination IP address tuple). Data center network traffic measurement is broadly categorized into active and passive measurements.
[0003] Active measurement typically involves proactively injecting specific network probe packets into the network traffic under test, and then analyzing the network traffic based on feedback from intermediate network nodes. Passive measurement, on the other hand, monitors network traffic without injecting new traffic. The injection process in active measurement may affect the efficiency and quality of network transmission, and the injected data packets may also be lost during transmission, leading to low measurement efficiency.
[0004] Existing passive reflow measurement schemes can be divided into counter-based methods and sketch-based methods. Counter-based methods (such as frequency, lossy counting, and space saving) primarily use a series of counters to store flow statistics and replace the recorded smallest flow with the newly arriving flow. As mentioned earlier, traffic in data center networks exhibits a skewed characteristic. That is, most network traffic is carried by only a small number of reflows, while the vast majority of other flows are small. Therefore, even if the stored smallest flow is actually a large flow, it will be incorrectly replaced by frequently arriving different smaller flows. Sketch-based methods are widely used in academia and industry. These methods use multiple hash functions to store flow information in a sketch, obtaining the desired recorded information through different replacement formulas and statistical methods, representing a streamlined data structure. Compared to counter-based methods, sketch-based methods significantly reduce the hardware memory footprint of the detection nodes. The sketch structure is typically deployed at intermediate network nodes (such as data center switches) and provides the recorded measurement results to locations that need this information (such as data center control plane nodes).
[0005] Existing passive measurement methods for reflow identification in data center networks include: software-based algorithms, such as the USS (Unbiased Space Saving) method, which involve complex processing steps and cannot be efficiently deployed on hardware (such as programmable switches and commercial switches), while slow processing speed leads to a decrease in network throughput; other works (such as CocoSketch) compromise on hardware implementation to simplify the processing flow, but this results in a decrease in the accuracy of reflow identification. UA-sketch incorporates information from consecutively arriving data packets into the sketch's statistical process to achieve a more accurate identification rate, but this increases hardware storage overhead and processing time overhead. Its insert process involves both multiplication and division, which is not conducive to the implementation of this scheme on programmable switches and will significantly increase hardware processing latency. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for identifying network reflow.
[0007] To solve the above-mentioned technical problems, the technical solution proposed by this invention is: a network reflow identification method, comprising the following steps: 1) In a programmable switch, two or more one-dimensional hash tables are arranged based on a sketch structure. Each hash table has w storage units. The data recorded in each storage unit includes a unique ID that identifies each flow and the flow size of that flow. 2) The programmable switch's processing program calls the hash function to calculate the index value of the current data packet in the corresponding hash table based on the ID of the current data packet's flow; 3) For each hash table, the current packet processor finds the packet's position in the table based on the index value. If no data is recorded at the current position, the packet's ID and payload bytes are initialized to the current position. If the ID recorded at the current position matches the ID of the current packet, the current packet's payload bytes are added to the current packet's throughput. If the ID recorded at the current position differs from the current packet's ID, a probability is calculated using a preset function to determine whether to replace the ID recorded at the current position. If replacement is performed, the current packet's throughput bytes are added to the current packet's throughput. 4) After step 3) is completed, update all hash tables; whenever the ID recorded at the current packet position in a hash table is the ID of the current packet, and the traffic size recorded at the current position in the hash table exceeds the set threshold, increment the counter carried by the packet handler during its lifetime to obtain a reflow hit count x. 5) Compare the number of recurrent flow hits x with the number of confidences x_alpha. If x is greater than or equal to x_alpha, then the flow to which the data packet belongs is considered a recurrent flow; at this time, the information of this flow from the control plane node is fed back; x_alpha is greater than or equal to 2. 6) After identifying a recurrent flow, the match-action unit rules of the programmable switch are updated through the control plane program to more accurately monitor recurrent flow information and improve the accuracy of identifying other recurrent flows in the future.
[0008] In the above-described network reflow identification method, preferably, after the reflow information in step 5) is fed back to the control plane node, the programmable switch table entry is updated; after the programmable switch table entry is updated, the ID of the reflow and the flow size recorded in the hash table are cleared to zero.
[0009] In the preferred embodiment of the network reflow identification method described above, the table update step of the programmable switch includes: using the flow identifier of the current data packet as a condition for the matching rule in the processing pipeline of the programmable switch to generate a new match-action unit; the matching rule is that whenever a match is successful, the payload bytes of this data packet are accumulated into a counter corresponding to this rule to record the current flow size, and then the data packet is directly forwarded without participating in the data update of the hash table of the Sketch structure; the flow identifier includes an IP 5-tuple or a source / destination IP tuple.
[0010] In the above-mentioned network reflow identification method, preferably, the calculation formula of the preset function is: p = S_packet / (S_packet + S_bucket), where S_packet is the payload bytes of the current data packet, S_bucket is the current location record traffic size, and p is the probability.
[0011] In the above-described network recurrent flow identification method, preferably, the programmable switch can query the ID of a single flow and its size, and query the IDs of all recurrent flows and their sizes. When querying the ID and size of a single flow, the operator initiates a query operation, reads the match-action unit table of the programmable switch, and checks if there is matching information for this flow. If so, the value of the corresponding counter is added to the previous threshold θ as the measured current flow size and fed back to the operator. If no matching information is found, the operator continues to query the sketch table, retrieves all recorded flow IDs that match the corresponding flow sizes, takes the average value as the measured current flow size and feeds it back to the operator. If no matching data is found in the sketch table, the operator is not informed that no record was made. When querying the IDs of all recurrent flows and their sizes, the operator initiates the query operation, reads the match-action unit table of the programmable switch, and returns the flow ID and the measured current flow size corresponding to each matching rule to the operator.
[0012] In the above-described network reflow identification method, preferably, multiple hash tables each correspond to different hash functions. The original flow IDs stored in the storage units of multiple hash tables are all replaced with feature values calculated by the feature hash function. Subsequently, when comparing the flow IDs of the storage units, the feature value of the current flow ID calculated by the feature hash function is compared with the feature value stored in the storage unit. The feature hash function includes the hash function corresponding to each of the multiple hash tables.
[0013] Compared with the prior art, the advantages of the present invention are as follows: the present invention designs a sketch structure of multiple hash tables in the programmable switch, and each flow responds to multiple hashes at the same time before performing subsequent identification operations, which improves the resistance to frequent small flow error replacement.
[0014] Compared with the USS method and cocoSketch method under the same storage size, the network reflow identification method of the present invention reduces the occurrence of frequent small flow error replacement by 90%. Attached Figure Description
[0015] Figure 1 This is a flowchart of the data packet processing after it arrives at the programmable switch in Example 1.
[0016] Figure 2 This is a flowchart of the reflow information query process in Example 1.
[0017] Figure 3 This is a system structure diagram in Example 1. Detailed Implementation
[0018] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.
[0019] It should be noted that when a component is described as being "fixed to, attached to, connected to or connected to" another component, it can be directly fixed to, attached to, connected to or connected to the other component, or it can be indirectly fixed to, attached to, connected to or connected to the other component through other intermediate connectors.
[0020] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.
[0021] This invention provides a method for identifying network reflows, comprising the following steps: 1) Construct a Sketch-like data structure. In the programmable switch, arrange two or more one-dimensional hash tables based on the sketch structure. Each hash table has w storage units. The data recorded in each storage unit includes a unique ID that identifies each flow and the flow size of that flow.
[0022] The arrival of a single data packet will drive a data update, similar to the USS and cocosketch methods, to ensure unbiased estimation of the data, as shown in steps 2 and 3).
[0023] 2) The programmable switch's processing program calls the hash function to calculate the index value of the current data packet's flow ID in the corresponding hash table; 3) For each hash table, the current packet's processing program finds its position in the table based on the index value. If no data is recorded at the current position, the packet's ID and payload bytes are initialized to the current position. If the ID recorded at the current position matches the ID of the current packet, the current packet's throughput is added to the current packet's payload bytes. If the ID recorded at the current position differs from the current packet's ID, a probability is calculated using a preset function to determine whether to replace the ID recorded at the current position. If replacement is performed, the current packet's throughput is added to the current packet's payload bytes. The preset function is calculated as: p = S_packet / (S_packet + S_bucket), where S_packet is the current packet's payload bytes, S_bucket is the current packet's throughput, and p is the probability.
[0024] 4) After step 3) is completed, update all hash tables; whenever the ID recorded at the position of the current data packet in a hash table is the ID of the current data packet, and the traffic size recorded at the position of the current hash table exceeds the set threshold, increment the counter carried by the current data packet's handler throughout its lifetime to obtain a reflow hit count x. 5) Compare the number of recurrent flow hits x with the number of confidences x_alpha. If x is greater than or equal to x_alpha, then the flow to which the data packet belongs is considered a recurrent flow. At this time, the information of this flow is fed back to the control plane node; x_alpha is greater than or equal to 2. 6) After identifying a recurrent flow, the match-action unit rules of the programmable switch are updated through the control plane program to more accurately monitor recurrent flow information and improve the accuracy of identifying other recurrent flows in the future.
[0025] In this invention, if a higher resistance to frequent small-flow erroneous replacement of recurrent flows is required, the value of x_alpha is set to be larger. A larger value of x_alpha indicates that the conditions required to identify recurrent flows are more stringent, and the resistance is stronger. In data center networks, there are many packets of recurrent flows, but the number of recurrent flows themselves is not large, and they are characterized by frequent and continuous arrival. Other small flows, on the other hand, are numerous in number but arrive sporadically in small quantities. Due to the large number of small flows, it is inevitable that small flows and large flows will be assigned to the same location in the hash table, resulting in erroneous replacement. In this case, for a single hash table, due to the frequent arrival of recurrent flows, the probability that their packets carrying flow IDs occupy the flow ID recorded in the current storage location is much greater than that of other small flows. Therefore, when a packet of a recurrent flow arrives, the probability that it triggers the confidence count is much greater than that of packets of other small flows. As a result, the recurrent information identified is significantly reduced in cases of erroneous replacement by small flows compared to other methods.
[0026] In this invention, after the reflow information in step 5) is fed back to the control plane node, it triggers an update of the programmable switch's table entries. After the programmable switch's table entries are updated, the ID and flow size of the reflow recorded at the corresponding position in the hash table are cleared. In this invention, when a reflow triggers an update of the programmable switch's table entries, the ID and flow size of the reflow recorded at the corresponding position in the hash table are cleared, thus freeing up space in the hash table and preventing interference with the identification of other flows.
[0027] In this invention, the table update step of the programmable switch includes: using the flow identifier of the current data packet as a condition for the matching rule in the processing pipeline of the programmable switch to generate a new match-action unit; the matching rule is that whenever a match is successful, the payload bytes of the data packet are accumulated into a counter corresponding to the rule to record the current flow size, and then the data packet is directly forwarded without participating in the data update of the Sketch table; the flow identifier includes an IP 5-tuple or a source / destination IP tuple.
[0028] like Figure 2 As shown, in this invention, the programmable switch can query the ID of a single flow and its size, and query the IDs of all recurrent flows and their sizes. When querying the ID and size of a single flow, the operator initiates a query operation, reads the match-action unit table of the programmable switch, and checks if there is matching information for this flow. If so, the value of the corresponding counter is added to the previous threshold θ as the measured current flow size and fed back to the operator. If no matching information is found, the operator continues to query the sketch table, retrieves all recorded flow IDs that match the corresponding flow sizes, takes the average value as the measured current flow size and feeds it back to the operator. If no matching data is found in the sketch table, the operator is not informed that no record was made. When querying the IDs of all recurrent flows and their sizes, the operator initiates the query operation, reads the match-action unit table of the programmable switch, and returns the flow ID and the measured current flow size corresponding to each matching rule to the operator.
[0029] In this invention, multiple hash tables each correspond to different hash functions. The original stream IDs stored in the storage units of multiple hash tables are replaced with feature values calculated by the characteristic hash function. These feature values are calculated using the characteristic hash function for the original stream ID. Subsequently, when comparing the stream IDs of the storage units corresponding to this original stream, the feature value of the stream ID calculated by the characteristic hash function is compared with the feature value stored in the storage unit. The characteristic hash function includes the hash functions corresponding to each of the multiple hash tables. That is, in this invention, feature values can be used instead of the stream IDs stored in the storage units of the hash tables, and consistency checks can be performed using these feature values, thereby saving storage space. It should be noted that the characteristic hash function can be any hash function besides the hash functions corresponding to the multiple hash tables.
[0030] During the initialization and periodic reconstruction of the Sketch table in this invention, the record IDs at all positions in multiple hash tables are set to null, and the record flow size is set to 0. Example
[0031] The system structure diagram of this embodiment is as follows: Figure 3 As shown, this is implemented using a programmable switch connected to the network under test, taking a programmable switch based on the P4 language as an example. The sketch table can be implemented using P4 language. The specific parameters of the sketch table are as follows: the number of hash tables d=2; the two hash function values are obtained by using bits 1-16 and bits 17-32 of the Cyclic Redundancy Check (CRC32) algorithm, respectively, each yielding an integer between 0 and 65535. The length of the hash table w is 65536. The flow ID is taken from the IP 5-tuple. In the Ingress code segment of the P4 language, the hash function is calculated after the current data packet arrives at the switch, and the corresponding ID and flow size data are updated. The threshold setting rules are set according to the current network traffic conditions.
[0032] The following example illustrates the process of triggering reflow detection and updating the match-action table after a data packet arrives. Figure 1 As shown, after a data packet arrives at the switch, the switch first parses it, extracting its IP 5-tuple as the identifier of the flow ID to which the data packet belongs, denoted as ID1. It also extracts the number of bytes of the payload carried by the data packet. Using the extracted flow ID as input, two hash functions are used to calculate the index and position of the current data packet in two hash tables. The number of bytes carried by the current data packet is added to the corresponding position in the two-layer hash tables, and the replacement probability of the current data packet in the two-layer hash tables is calculated according to p = S_packet / (S_packet + S_bucket). This probabilistic replacement step aims to give later-arriving recurrent flows a chance to be identified by the system, thus helping the subsequent separation into the switch's match-action unit for accurate identification. In this embodiment, the current data packet belongs to the flow ID ID1. The value recorded in the corresponding position of the current data packet in the two-layer hash table is currently (ID1, 10880 bytes) (ID2, 10400 bytes). Since the current data packet carries 40 bytes, the IDs in the first-layer hash table are consistent and no replacement is needed; the recorded value becomes (ID1, 10920 bytes). The replacement probability corresponding to the second-layer hash table is 40 / 10440. Because this flow arrives frequently at this time, the replacement probability is hit this time, and a replacement occurs, with the recorded data becoming (ID1, 10440 bytes). Assuming the current threshold is 10240 bytes, the recorded ID is consistent with the flow ID of the current data packet, and the number of rows where the flow reaches the threshold is 2, reaching a confidence count of 2. In this embodiment, the confidence count is set to 2. The processing program is triggered to inform the control plane node of the flow information, and then the table update step of the programmable switch is triggered, forming a matching rule for ID1 and updating the match-action table. The control plane code implements the generation of new matching rules from the reported flow information and sends them to the match-action unit of the programmable switch. When subsequent packets from this flow enter the switch's processing flow, they will first be captured by the matching rules and counted, and then forwarded directly without participating in the data update of the sketch table.
[0033] like Figure 2 As shown, during a query, control plane software can be written to obtain the corresponding data through the programmable switch interface to complete the query process. The aging and deletion of matching entries can be implemented through the aging settings of the programmable switch.
[0034] In this embodiment, a sketch structure of multiple hash tables is designed in the intermediate nodes of the network. Each flow responds to multiple hashes simultaneously before performing subsequent identification operations, which improves the resistance to frequent small flow erroneous replacements.
[0035] Compared with the USS method and cocoSketch method under the same storage size, the network reflow identification method of the present invention reduces the occurrence of frequent small flow error replacement by 90%.
Claims
1. A network reflow identification method, characterized by; Includes the following steps: 1) In a programmable switch, two or more one-dimensional hash tables are arranged based on a sketch structure. Each hash table has w storage units. The data recorded in each storage unit includes a unique ID that identifies each flow and the flow size of that flow. 2) The programmable switch's processing program calls the hash function to calculate the index value of the current data packet's flow ID in the corresponding hash table; 3) For each hash table, the current packet's handler finds its position in the table based on the index value. If there is no record at the current position, the packet's ID and payload bytes are initialized to the current position. If the ID recorded at the current position matches the ID of the current packet, the packet's payload bytes are added to the current packet's payload size. If the ID recorded at the current position is different from the current packet's ID, a probability is calculated using a preset function to determine whether to replace the ID recorded at the current position. If it is replaced, the packet's payload bytes are added to the current packet's payload size. 4) After step 3) is completed, update all hash tables; whenever the ID recorded at the position of the current data packet in a hash table is the ID of the current data packet, and the traffic size recorded at the position of the current hash table exceeds the set threshold, increment the counter carried by the current data packet's handler throughout its lifetime to obtain a reflow hit count x. 5) Compare the number of recurrent flow hits x with the number of confidences x_alpha. If x is greater than or equal to x_alpha, then the flow to which the data packet belongs is considered a recurrent flow. At this time, the information of this flow is fed back to the control plane node; x_alpha is greater than or equal to 2. 6) After identifying the recurrent flow, update the match-action unit rules of the programmable switch through the control plane program.
2. The network reflow identification method according to claim 1, characterized in that: After the reflow information in step 5) is fed back to the control plane node, the programmable switch table entry update is triggered; after the programmable switch table entry update is triggered, the ID of the reflow and the flow size recorded in the hash table are cleared to zero.
3. The network reflow identification method according to claim 2, characterized in that: The table update steps of the programmable switch include: using the flow identifier of the current data packet as a condition for the matching rule in the programmable switch processing pipeline to generate a new match-action unit; the matching rule is that whenever a match is successful, the number of payload bytes of this data packet is accumulated into a counter corresponding to this rule to record the current flow size, and then the data packet is directly forwarded without participating in the data update of the hash table of the Sketch structure; the flow identifier includes an IP 5-tuple or a source / destination IP tuple.
4. The network reflow identification method according to claim 1, characterized in that: The calculation formula for the preset function is: p = S_packet / (S_packet + S_bucket), where S_packet is the payload bytes of the current data packet, S_bucket is the current location record traffic size, and p is the probability.
5. The network reflow identification method according to claim 2, characterized in that: The programmable switch can query the ID of a single flow and its size, as well as the IDs of all recurrent flows and their sizes. When querying the ID and size of a single flow, the operator initiates a query operation, reads the match-action unit table of the programmable switch, and checks if there is matching information for this flow. If so, the value of the corresponding counter is added to the previous threshold θ as the measured current flow size and fed back to the operator. If no matching information is found, the operator continues to query the sketch table, retrieves all recorded flow IDs that match the corresponding flow sizes, takes the average value as the measured current flow size and feeds it back to the operator. If no matching data is found in the sketch table, the operator is not informed that no record was made. When querying the IDs of all recurrent flows and their sizes, the operator initiates the query operation, reads the match-action unit table of the programmable switch, and returns the flow ID and the measured current flow size corresponding to each matching rule to the operator.
6. The network reflow identification method according to claim 1, characterized in that: Each of the multiple hash tables corresponds to a different hash function. The original stream IDs stored in the storage units of the multiple hash tables are replaced with the feature values calculated by the feature hash function. When comparing the stream IDs of the storage units later, the feature hash function is used to calculate the feature value of the current stream ID and compare it with the feature value stored in the storage unit. The feature hash function includes the hash function corresponding to each of the multiple hash tables.
Citation Information
Patent Citations
Network flow identification method based on Hash collision compensation
CN102025563A
Cooperative large flow detection method for programmable network
CN113132180A
