Dynamic network flow remapping based on feedback
By analyzing feedback data in the network and remapping high congestion flows, modifying its header fields to index new paths, the congestion problems caused by flow conflicts and link oversubscription in the network are solved, and dynamic load balancing and congestion reduction of the network are achieved.
Patent Information
- Application Number
- CN202380036278.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-18
- Filing Date
- 2023-02-19
- Publication Date
- 2024-12-06
AI Technical Summary
Congestion problems in existing networks due to conflicts between streams and oversubscription of links lead to packet delay or discarding, and the prior art is difficult to effectively reduce network congestion.
By analyzing network feedback data, select streams with high relative congestion levels for remapping, modifying their header fields and generating hash values to index to available paths, allowing the stream to follow a new path through the network, thereby reducing congestion.
Significantly reduce network congestion, dynamic load balancing, avoid congestion caused by excessive link ordering, and does not require changing the network structure. All remapping logic can be implemented at network endpoints, with little additional overhead.
Smart Images

Figure CN119096528A_ABST
Abstract
Description
Background Art
[0001] A computer "network" is a collection of interconnected or interlinked computing devices that can communicate with each other by exchanging data and by sharing resources. A network includes any number of endpoints. These endpoints are connected to each other via any number of switches. One endpoint (e.g., a "source endpoint") can send information to another endpoint (e.g., a "destination endpoint") in the form of a "packet."
[0002] It may be the case that multiple packets share common characteristics. As a result of these packets sharing these common characteristics, these packets may ultimately follow the same path through the network. The term "path" refers to a set of one or more specific switches and links (e.g., multiple links may connect pairs of switches) that are used to route packets through the network. When a set of packets with similar characteristics ultimately follow the same path, the set of packets may be referred to as a "flow." Packets belonging to the same TCP / IP session are included in a flow, and it is usually (although not always) the case that these packets follow the same path through the network.
[0003] Sometimes, flows in a network conflict with each other. For example, a first flow may follow a first path that includes switch "C". A second flow may follow a second path that also includes switch "C". If the bandwidth capacity of switch "C" is not high enough, switch "C" may become congested. Additionally or alternatively, congestion may occur as a result of suboptimal path selection through the network. Oversubscription of links may also cause congestion problems. Regardless of the cause, congestion may have a detrimental effect on the network, such as packet delays or even packet drops. Because flows often conflict with each other (e.g., by oversubscribing commonly used switches or links), it is desirable to implement a system that can help reduce congestion and conflicts in the network.
[0004] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is merely provided to illustrate one exemplary technology area in which some embodiments described herein may be practiced. Summary of the invention
[0005] Embodiments disclosed herein relate to systems, networks, devices, and methods for triggering flows to follow new paths through a network. The network is configured to provide multiple paths from one endpoint to another. A flow includes multiple network packets with similar characteristics. As a result, switches in the network route these network packets through the same path. By triggering flows to follow new paths, an attempt is made to reduce congestion in the network. As used herein, references to a "switch" may also include references to output ports and / or links of the switch. Therefore, the term "switch" should be broadly interpreted to include the switch itself, and additionally or alternatively include output ports of the switch and possibly even links to or from the switch.
[0006] Some embodiments identify multiple flows traversing a network. Embodiments analyze network feedback data to determine that flows conflict with each other and congest one or more links of the network. Embodiments select a particular flow to be remapped. Thus, the particular flow will be converted from following a first path through the network to following a second path through the network. In some embodiments, the process of selecting a particular flow is based on a remapping probability function that determines which flow to remap based on relative congestion levels for the flows.
[0007] The remapping process for a particular flow includes performing a plurality of operations for each new network packet in the particular flow. One operation includes selecting a field within a header of each new network packet. Another operation includes modifying a value in the selected field. After modifying the value, another operation includes performing a hash on the field value in the header (including the modified value described above) to generate a hash value for each new network packet. Yet another operation includes indexing the hash value into an available path for each new network packet, where the available path is a second path. The embodiment then causes the new network packet of the particular flow to follow the remapped second path through the network.
[0008] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0009] Additional features and advantages will be set forth in the following description and, in part, will be apparent from the description or may be learned by practicing the teachings herein. The features and advantages of the present invention may be realized and obtained by the components and combinations particularly pointed out in the appended claims. The features of the present invention will become more fully apparent from the following description and the appended claims or may be learned by practicing the present invention as set forth below. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments that are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments and are therefore not to be considered limiting of scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:
[0011] Figure 1 An example network is illustrated that includes multiple endpoints, multiple switches, and multiple paths between each endpoint.
[0012] Figure 2 Details of an example network packet are illustrated.
[0013] Figure 3 Details related to the flow are illustrated.
[0014] Figure 4 It illustrates how to determine the path for a packet.
[0015] Figure 5 An example scenario is illustrated where multiple flows conflict with each other on a switch, causing network congestion. Figure 5 A scenario is shown where multiple flows are hashed in such a way that the resulting paths selected for the flows end up using the same links, resulting in link conflicts.
[0016] Figure 6 Illustrated are examples of feedback data that may be received to determine whether a switch or a link between switches is overutilized.
[0017] Figure 7 An example process for triggering remapping of a flow is illustrated.
[0018] Figure 8 An example of a network in which flows have been remapped to avoid congestion is illustrated.
[0019] Fig.9A and Fig. 9B A flow chart of an example method for triggering a flow to follow a new path through a network is illustrated.
[0020] Fig.10 An example computer system is illustrated that may be configured to perform any of the disclosed operations. DETAILED DESCRIPTION
[0021] Embodiments disclosed herein relate to systems, networks, devices, and methods for triggering a flow to follow a new path through a network. A flow includes multiple network packets with similar characteristics. As a result, switches in the network route the network packets through the same path. By triggering a flow to follow a new path, congestion in the network is attempted to be reduced.
[0022] Some embodiments identify flows traversing the network. Embodiments analyze network feedback data to determine that flows conflict with each other and cause congestion in one or more links and / or switches. Embodiments select a particular flow to be remapped. Thus, the particular flow will be converted from following a first path through the network to following a second path. In some embodiments, the selection process is based on a remapping probability function that determines which flow to remap based on relative congestion levels for the flows.
[0023] The remapping process for a particular flow includes performing a plurality of operations for each new network packet in the particular flow. One operation includes selecting a field within a header of each new network packet. Another operation includes modifying a value in the selected field. After modifying the value, another operation includes performing a hash on the field value in the header (including the modified value described above) to generate a hash value for each new network packet. Yet another operation includes indexing the hash value into an available path for each new network packet, where the available path is a second path. The embodiment then causes the new network packet of the particular flow to follow the remapped second path to traverse the network.
[0024] Examples of technology benefits, improvements and real-world applications
[0025] The following section summarizes some example improvements and practical applications provided by the disclosed embodiments. However, it should be understood that these are only examples and the embodiments are not limited to these improvements.
[0026] The disclosed embodiments bring many benefits, advantages and practical applications to the technical field of network management. Specifically, the disclosed principles can be followed to beneficially reduce congestion in a network. When multiple flows use the same link and / or switch in a network, the link and / or switch may become oversubscribed or congested. When a link and / or switch operates at or even exceeds its bandwidth capacity, packets may be delayed or potentially dropped. Embodiments can detect when a network is congested and can also reroute or remap flows to avoid congestion.
[0027] The disclosed techniques can adapt to changing network conditions and can dynamically load balance flows to significantly or even completely eliminate congestion due to link oversubscription. The disclosed techniques are lightweight and do not require any network structure changes, as all remapping logic can be implemented in network endpoints.
[0028] Advantageously, the disclosed principles can be practiced using very little additional overhead imposed on the network. That is, an embodiment is configured to selectively modify a value in a header of a packet. By modifying the value, the packet will then travel through the network along a new path, which will be described in more detail later. Other packets in the stream can be similarly modified. By making this modification, significant improvements to the functionality of the network can occur (e.g., by changing how packets are sent across different available paths), and a significant reduction in congestion can be achieved. Therefore, these and many other benefits will now be described in more detail throughout the remainder of this disclosure.
[0029] Example Network
[0030] Now will focus on Figure 1 , Figure 1 An example network 100 is illustrated that may be used to implement the disclosed principles. Without limitation, network 100 may be any type of network.
[0031] Network 100 is shown including several endpoints, such as endpoints 105, 110, 115, and 120. As used herein, the term "endpoint" refers to a computing device that communicates with one or more intermediate switches and links to communicate with another endpoint. Figure 1 A number of other endpoints are shown that are not labeled. Network 100 may include any number of endpoints.
[0032] The network 100 is also shown to include several switches, such as switches 125, 130, 135, 140, 145, 150, and 155. As used herein, the term "switch" refers to a computing device that forwards network data packets between different devices (such as between switches and between switches and endpoints). Each switch in the network 100 is structured to support a specified amount of bandwidth (e.g., it can be assumed that the switch can handle full line rate throughput on all its ports, so that they do not have any inherent bottlenecks when there is no output contention), the full line rate throughput is the maximum amount of data that the switch can handle, and more specifically, the full line rate throughput is the maximum amount of throughput that can be passed through the switch. In other words, bandwidth generally refers to the speed at which the switching device itself can switch when routing data between different ports.
[0033] Figure 1 An example scenario is shown in which endpoint 105 is using network 100 to transmit packet 160 to another endpoint in the network. As used herein, the term "packet" generally refers to a unit of data that is routed between multiple devices. Typically, a single packet is a small fragment that forms a larger unit of data. Figure 2 Some additional details are provided.
[0034] Figure 2 A network packet 200 is shown, which represents a network packet from Figure 1 The network packet 200 typically includes control information 205 and user data or payload 210.
[0035] The control information 205 may include various header fields 215, and more specifically, fields in the header of the network packet 200. One example of a field is a source network address 220, which refers to the address of the originating or source endpoint of the network packet 200. Typically, the source network address 220 includes fields for an Internet Protocol (IP) address 225 for the source and a source port 230 from which the network packet 200 is transmitted.
[0036] Another field in the control information 205 is the destination network address 235. Typically, the destination network address 235 includes fields for the IP address 240 of the destination endpoint and the destination port 245 where the network packet 200 is to be received.
[0037] Control information 205 also typically includes information about the protocol 250 used to transmit network packet 200. Examples of different network protocols include, but are certainly not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP) protocol, Border Gateway Protocol (BGP), User Datagram Protocol (UDP), etc.
[0038] The combination of source IP address 225, source port 230, destination IP address 240, destination port 245, and protocol 250 is generally referred to as 5-tuple 255. That is, 5-tuple 255 generally refers to a set of five values that form a TCP / IP connection.
[0039] The control information 205 also typically includes an error detection code 260, sequencing information 265, and an interface field 270. The error detection code 260 is an additional bit set that allows the receiver to determine whether the network packet has been corrupted during transmission. The sequencing information 265 can be used to determine which order the network packets should be organized into to reconstruct the message. Sometimes, the packets are received in an order different from the order in which they were sent, and the sequencing information 265 can be used to reorder the packets. The interface field 270 records both the ingress and egress information for the packet (i.e., the direction of travel of the network packet).
[0040] Return to Figure 1 , typically global visibility 165 between endpoints is not available. That is, there may be multiple endpoints in the network. Typically, global visibility is not provided to multiple endpoints. Global visibility 165 refers to the ability of endpoints to communicate directly with each other and crosstalk.
[0041] Network Flow
[0042] Now will focus on Figure 3 , Figure 3 Shown with Figure 1 The example network 300 is similar to the network 100. The network 300 includes endpoints 305, 310, 315, and 320. The network 300 also includes switches 325, 330, 335, 340, and 345.
[0043] Endpoint 305 is using network 300 to transmit packet 350 to endpoint 310. Similarly, endpoint 320 is using network 300 to transmit packet 355 to endpoint 315.
[0044] Typically, the packet sets share some common characteristics. For example, although the effective loads of different network packets can be different, the control information of these network packets can be substantially the same. For example, the source network address, the destination network address, the protocol, the error detection code, and even the interface field can be quite similar or even identical to each other. Because these network packets all have common characteristics, typically these packets will be transmitted through the network 300 via the same path 360. When the group of packets has similar characteristics and follows the same path through the network, the group of these packets is referred to as a "stream". That is, in some cases, the similar characteristics of the network packets include similar header characteristics, which make the network packets included in the same stream.
[0045] Figure 3 A first flow 365 is shown. Flow 365 includes packet 350, among many other packets. The solid bold arrows indicate the path of flow 365 through network 300. For example, flow 365 originates at endpoint 305 and then travels through switch 325, switch 330, and switch 335 before reaching destination endpoint 310.
[0046] Figure 3 A second flow 370 is shown. Flow 370 includes packet 355, among many other packets. The bold dashed arrows represent the path of flow 370 through network 300. For example, flow 370 originates at endpoint 320 and then travels through switch 340, switch 345, and switch 335 before reaching destination endpoint 315. Thus, a flow is defined as a group of network packets that share common characteristics, where as a result of these common characteristics, the packets in the flow tend to follow the same path through the network.
[0047] Different techniques can be used to route packets through a network. One common technique is called equal cost multipath or ECMP. Figure 4 ECMP is described in more detail in .
[0048] Specifically, each network packet has a set of packet characteristics 400. These characteristics include Figure 2 The ECMP 405 process includes information related to the control information 205 of the packet and sometimes includes information related to the user data / payload 210. The ECMP 405 process involves the network switch extracting the packet characteristics 400 and then performing a hash 410 operation on these characteristics. The resulting hash is then mapped or indexed to the next device or switch that will be used to transmit the packet (more specifically, the resulting hash is mapped to the output port of the device or the next link). The ECMP 405 process is repeated at each hop 415 of the network. By each hop, it means that each receiving switch (i.e., the switch that receives the network packet) performs the ECMP 405 hash process. Ultimately, the path 420 through the network is determined, and the packet is delivered to the destination endpoint.
[0049] Data center networks typically provide multiple paths between any given pair of endpoints. The industry standard mechanism for load balancing flows across these multiple paths is ECMP (Equal Cost Multipath), which applies a hash function to a subset of packet header fields (usually a "5-tuple") to determine the output port on each switch along the packet's path. The advantage of this routing strategy is that it guarantees in-order delivery because packets belonging to the same flow will always take the same path (because they have the same header bits and will always take the same output port on each switch hop along the path). However, a disadvantage of ECMP is that the hashing mechanism is oblivious to the load across multiple candidate paths and underlying links, which can lead to network bottlenecks and congestion due to link oversubscription when the aggregate throughput demand of flows sharing a link exceeds its capacity. An example will be helpful.
[0050] about Figure 3 , endpoint 305 transmits packet 350 to switch 325. Switch 325 then performs the ECMP process by extracting various features or characteristics from packet 350 (e.g., perhaps by extracting 5-tuple information). These features are fed into a hash function to generate a resulting hash. The hash is then mapped or indexed to the next available switch in the network (more specifically, it is mapped to an output port or link). The routing table for switch 325 reflects the next link or switch to be used. The routing table does not need to be updated; instead, an embodiment is configured to direct traffic on another valid route or path that already exists. Here, the result of the hash is mapped or indexed to switch 330, so packet 350 is routed from switch 325 to switch 330. Switch 330 is a new "hop" in the network.
[0051] Switch 330 then performs the same ECMP process. The result of the hash is mapped to switch 335. The routing table for switch 330 is then updated to include this information. Thus, the ECMP 405 process typically includes obtaining control information from the packet (e.g., such as from Figure 2 The information is extracted from the 5-tuple 255 of the packet and a hash is performed on the information set. The resulting hash is mapped to the next switch that will be used to transmit the packet. This process can be repeated for each hop through the network. The combination of these hops is the path that the packet travels through the network.
[0052] Figure 5 An example network 500 is shown that represents the network discussed so far. Here, the network 500 is shown to include two different flows, namely, flow 505 and flow 510. As a result of performing the ECMP process on the packets in each flow, both flows are shown to be routed through the same switch 515. Because both flows 505 and 510 are routed through the same switch 515 (and specifically through the same output port or link), the network 500 experiences congestion 520 at the switch 515 (and specifically at a specific output port or link). Normally, different flows passing through the same switch do not cause congestion problems. However, when these flows utilize the same output port or link, link conflicts and congestion will emerge if the bandwidth of the link is burdened or overused by multiple flows. For example, if the input and output ports of the switch can handle 100GB, traffic from two input ports of 100GB to the same output port of 100GB will cause congestion. Congestion or oversubscription occurs when switches in a network operate near (e.g., within a threshold level) or possibly even above their bandwidth capabilities. When congestion occurs, the buffers of the switches will begin to fill up, causing delays and potentially even packet losses.
[0053] The disclosed embodiments advantageously provide techniques for "remapping" or "rerouting" flows so that the flows follow different paths through the network. By doing so, the disclosed embodiments can advantageously reduce or even eliminate network congestion.
[0054] Receive feedback to determine if the network is congested
[0055] Figure 6A congestion 600 scenario involving a first switch 605 (having an output port 605A) and a second switch 610 is shown. These switches represent the switches illustrated in the network of the previous figures. A link 615 is shown between the two switches 605 and 610. The principles disclosed herein are primarily triggered when a link utilization 620 between the switches exceeds a utilization threshold, such that the link 615 and the switches 605, 610 are determined to be congested or oversubscribed 625. "Link utilization" generally refers to the amount of bandwidth used by a switch to transmit data packets. For example, a switch may have a bandwidth of 100GB. If the switch is being used for capacity (e.g., 100GB), then the switch may be said to be oversubscribed.
[0056] "Oversubscription" refers to the overall switching and link bandwidth of a particular switch and / or link being less than the available bandwidth or aggregate bandwidth available on all ingress switch ports. To determine whether a switch and / or link is oversubscribed, an embodiment receives (e.g., at an endpoint) feedback 630 from the network, and specifically feedback 630 from various switches and / or links in the network.
[0057] like Figure 6 As shown, feedback 630 may include one or more of round trip delay 635 metric, explicit congestion notification (ECN) marking 640, effective throughput 645 metric, buffer status 650, information or congestion notification packet (CNP) 655 metric. Ellipses 660 show that other parameters may also be included in feedback 630.
[0058] The round trip delay 635 (or round trip delay (RTD) or round trip time (RTT)) metric refers to the amount of time it takes for a packet to be sent to an endpoint plus the amount of time it takes for a receipt confirmation to be returned. The ECN marking 640 is an end-to-end notification that indicates congestion by marking or marking a packet when it is delayed to indicate congestion. The effective throughput 645 metric provides a measurement indicating the speed at which data travels through the network. The buffer status 650 information indicates how full the buffer is, and can also indicate how long a packet resides in the buffer. The CNP 655 metric can also be used to indicate congestion in a network.
[0059] Various switches in the network can provide this feedback 630 to one or more endpoints in the network. The endpoints can then use the feedback 630 to trigger a remapping process that causes the flow to follow a different path through the network. Therefore, the present disclosure will now turn to a discussion of the remapping process.
[0060] Remapping flows to reduce congestion
[0061] Figure 7An example process flow 700 is shown that may be followed to remap flows to reduce or alleviate congestion in a network. Initially, the process flow 700 includes a step 705 of obtaining network metrics such as network feedback data 710. The network feedback data 710 represents Figure 6 The feedback mentioned in 630. The endpoint can receive network metrics from various switches in the network.
[0062] Based on the feedback, the endpoint may determine that the network is congested. As a result, the endpoint triggers step 715, in which packets of the flow are modified. That is, endpoint 720 triggers packet modification step 715.
[0063] As previously described, the ECMP process performs hashing on various field values in the control information of the packet (e.g., which may be a 5-tuple of information). In accordance with the disclosed principles, an embodiment selects a field in the control information to modify the value of the field. Beneficially, an embodiment selects a field whose modification will not change the end destination of the packet. As an example, an embodiment may modify the source port for a packet. Modifying the value in the source port field will have no influence or effect on the location where the packet will be delivered.
[0064] However, by modifying the value, the resulting hash will be different from its original value. Because the resulting hash will be different, the hash will be mapped or indexed to a new switch (more specifically, to an output port or link), causing the packet to follow a different path through the network, and thereby causing the rest of the flow to also pass through the new / different path through the network. Therefore, in order to remap the flow so that the flow follows a different path through the network, the endpoint is configured to change the value in the header of the packet of the flow. When the switches perform their corresponding hash operations, the modification of the header value will cause subsequent packets in the flow to pass through the network through the new path. By causing the flow to follow a new path, an embodiment can beneficially redirect or remap the flow so that congestion is avoided. Further details on this aspect will be provided later.
[0065] The determination of which flow to remap may optionally be based on a remap probability function 725. In terms of simplicity, the simplest remapping option would be to remap each flow each time congestion is detected in the network. However, this option sometimes causes problems. For example, if each flow were to be remapped when congestion was detected, the remapping may end up contributing more to the congestion problem.
[0066] As an analogy, consider a scenario where drivers are driving on the interstate. In response to an upcoming construction zone, everyone's GPS reroutes to use the same detour. In this scenario, the detour route will now be highly congested and the rerouting does not solve this problem. Therefore, it is beneficial to employ some intelligence in selecting which flows to remap. Similar to this analogy, the disclosed embodiments employ intelligence in selecting which flows to reroute in an effort to avoid additional conflicts, congestion, or oversubscription scenarios. Note that embodiments can be restrictive when rerouting or remapping occurs so that the remapping process is not overly aggressive.
[0067] Beneficially, some embodiments attempt to categorize different flows into different buckets based on how severe the congestion is for each of these different flows. Prioritization based on congestion levels can then be used to determine whether a flow is to be remapped.
[0068] For example, embodiments may examine various metrics, including normalized congestion notification packets (CNPs). When a sender endpoint receives these CNPs, the sender endpoint can keep track of how many CNPs have been received within a time window. The sender endpoint may also normalize the CNPs based on how many packets the sender endpoint has sent.
[0069] For example, if a sending endpoint is to send 1,000 packets and is to receive CNP for 100 of those packets, the endpoint may generate a table illustrating the probability that a flow should be remapped based on the determined congestion (as determined by the relationship between the sent packets and the detected CNP).
[0070] The idea here is that endpoints don't necessarily want to treat all flows uniformly, because if some links are oversubscribed (e.g., perhaps over 800% or some other value), flows going through those links should have a higher probability of being remapped, because those flows may be the ones that are truly suffering. In contrast, links that are only slightly oversubscribed do not need to have their corresponding flows remapped as aggressively as flows going through links that are severely oversubscribed. Therefore, flows can be selectively remapped based on how much their transmission rates are affected. The remapping probability function can measure or determine how poorly flows are performing through the network. The remapping probability function can then be used to prioritize the remapping of flows in an attempt to reduce congestion and provide flows with a faster path through the network.
[0071] In this sense, the remapping probability function does not necessarily produce a strict prioritization scheme. As an example, if an endpoint has multiple flows, an embodiment can trigger the remapping of only the first flow from the endpoint, leaving the other flows in their original state. Even if the multiple flows conflict initially, remapping one flow (although the multiple flows may conflict with each other) can potentially cause congestion reduction. Therefore, an embodiment can be selective in determining which flows to remap.
[0072] Depending on traffic characteristics and reaction time requirements, the mapping iteration length may also be independent of the observation window duration and may range from a few microseconds (e.g., on the order of network RTT) up to a few seconds. The remapping decision is probabilistic and may be related to both absolute metrics (e.g., effective throughput threshold) and relative metrics (e.g., the flow with the highest normalized CNP received within the observation window) that are fed into the function that generates the remapping probability.
[0073] In its simplest form, the remapping probability function can be a table that maps ranges of a given metric to flow remapping probabilities (e.g., flows with average RTTs from 20us to 60us can be remapped with a probability of 5%). The remapping probability function can also be adjusted over time based on network conditions (e.g., in the case of a table-based probability mapping function, the table can be adjusted to initially focus only on remapping flows with high oversubscription, and gradually begin remapping flows with less severe oversubscription as overall network congestion improves).
[0074] In any case, when a flow is selected for remapping, an embodiment may perform step 730: determining or affecting a new path for the flow. This process is performed by an endpoint modifying one of the fields in the packet of the flow. When the packet arrives at the switch, the switch (e.g., as shown by switch 735) performs a hash operation on the set of values in the header of the packet. In some cases, hashing the field values in the header is performed using an equal cost multi-path (ECMP) hash function.
[0075] The resulting hash is then mapped or indexed (as shown by index 740) to the next switch in the network (more specifically, the next output port or link). Because the hash is different from the previous hash (as a result of performing the hash on the new input data set), the path that the packet will follow will be different from the earlier path used by the earlier packet. Each switch that receives the packet will perform its own corresponding hash operation to select the next switch to receive the packet. Performing the hashing on the field values in the header after the values in the selected fields have been modified results in the random generation of a new / second path. As a result, the second path is not the pre-selected path.
[0076] When packets in the flow are subsequently modified by the endpoint, the flow will begin to traverse a new path through the network. Step 745 reflects the scenario where the flow begins a new route or a new path through the network. After the new network packets of the flow are caused to follow the remapped second path, new network feedback data is received and used to determine whether the network is still congested.
[0077] As shown by repetition 750, process flow 700 may be repeated until a convergence threshold 755 is reached. Convergence threshold 755 may reflect the level of conflicts or congestion present in the network. If the number of conflicts is below convergence threshold 755, process flow 700 may end. On the other hand, if (even after the initial remapping event) convergence threshold 755 is not met even after a predetermined period of time after the remapping event (in order to allow the network to stabilize), process flow 700 may be repeated and another remapping event may be triggered. In some embodiments, there is no limit to the number of times a flow may be remapped.
[0078] Figure 8 Shows the Figure 5 The result of remapping one of the streams shown. Specifically, Figure 8 Network 800 is shown including endpoints 805, 810, 815, and 820. Network 800 also includes switches 825, 830, 835, 840, and 845. Endpoint 805 transmits packets, such as packet 850, in the form of stream 855. Similarly, endpoint 820 transmits packets, such as packet 860, in the form of remapped stream 865. Remapped stream 865 is from Figure 5 865. The embodiment of the present invention is a remapped version of the flow 510 of the present invention. Previously, flows 505 and 510 conflicted with each other at switch 515. In response to detecting the conflict via network feedback data, endpoint 820 triggers a remap event for flow 510, causing the flow to subsequently follow a new path represented by the remapped flow 865. The flow is remapped by endpoint 820 modifying the header value for each new packet that is part of the flow. As a result of the modification, the switch generates a new hash result, which causes the flow to follow a new path through the network. Therefore, by modifying the value in the header of the packet, the embodiment is able to remap or reroute the entire flow.
[0079] Example Method
[0080] The following discussion now relates to several methods and method actions that can be performed. Although method actions may be discussed in a particular order or illustrated in a flowchart as occurring in a particular order, a particular order is not required unless specifically stated or because an action depends on another action being completed before the action is performed.
[0081] Now turn your attention to Fig.9A and Fig. 9B , Fig.9A and Fig. 9B A flow chart of an example method 900 for triggering a flow to follow a new path through a network is illustrated. Note that a flow includes multiple network packets with similar characteristics such that switches in the network route the multiple network packets through the same path. By triggering a flow to follow a new path, an attempt is made to reduce congestion in the network. Method 900 can be performed by an endpoint as a source for a flow and by various switches that route the flow through the network.
[0082] Method 900 includes an act (act 905) of identifying (eg, by an endpoint) a plurality of flows traversing a network. For example, Figure 5 Flows 505 and 510 may be identified. In some cases, the flows originate from the same endpoint.
[0083] Action 910 includes analyzing network feedback data (e.g., by an endpoint) to determine that multiple flows conflict with each other and congest one or more links of the network. For example, an endpoint in the network (e.g., a starting endpoint for a flow) can receive feedback data from a switch in the network. The endpoint can analyze the data to determine whether the flows conflict with each other at a particular switch. The network feedback data can optionally include one or more of the following: round trip delay for a network packet, an explicit congestion notification (ECN) marking on a network packet, an effective throughput metric, a buffer status, or a congestion notification packet (CNP) metric.
[0084] Action 915 includes selecting (e.g., by the endpoint) a particular flow included in the plurality of flows to be remapped so that the particular flow will be converted from following a first path through the network to following a second path through the network. In some cases, the process of selecting the particular flow is based on a remapping probability function that determines which flow in the plurality of flows to remap based on relative congestion levels for the plurality of flows.
[0085] Remapping a specific stream includes Fig. 9B For example, Fig. 9B The actions in are performed for each new network packet in a particular flow. For illustration purposes, action 915A includes selecting (e.g., by an endpoint) a field within a header of each new network packet. The selected header is a header whose modification will not cause the packet to be sent to a new destination endpoint. For example, it is often the case that a source port value can be changed because changing the value will not affect the destination location of the packet. In this way, the selected field can include a source port value.
[0086] Action 915B includes (e.g., by an endpoint) modifying a value in a selected field. The modification of the value may be performed in any manner. As previously described, modifying a value in a selected field may include modifying a specific value, and the modification of a specific value does not cause a change in where a specific flow will be delivered.
[0087] After modifying the value, action 915C includes (e.g., by one or more switches) performing a hash on the field values in the header (including the modified value) to generate a hash value for each new network packet. As described above, this action can be performed by each switch in the network that receives the packet. In some cases, the process of performing a hash on the field values in the header can include performing a hash on the following fields (more specifically, the values in these fields): a source network address field (which can include both a source IP address and a source port value), a destination network address (which can include both a destination IP address and a destination port value), and a protocol.
[0088] Action 915D includes indexing (e.g., by one or more switches) the hash value into an available path for each new network packet, where the available path is a second path. The second path is typically different from the first path in that the second path typically includes at least one switch that is not included in the first path. The phrase "indexing the hash value into an available path" typically means that the hash is indexed into a subsequent switch, which then performs another hash operation, the hash resulting from the hash operation is then indexed into another switch, and so on, until the packet reaches the destination endpoint.
[0089] Return to Fig.9A , action 920 includes causing (e.g., by one or more switches) a new network packet of a particular flow to follow the remapped second path through the network. That is, the endpoint modifies the header fields of the packet. The packet is then sent to a first switch in the network. From there, the packet (and therefore the flow) will follow the new path because the switch will select the new path based on the modified value in the header. Method 900 can be repeated, or more precisely, iterable, until a convergence threshold is met.
[0090] The disclosed functionality is available when there are multiple paths between a source endpoint and a destination endpoint. For example, both a first path and a second path may be used to route a particular flow from a source endpoint to a destination endpoint. In some cases, a third path may also be used to route a particular flow from a source endpoint to a destination endpoint. There may be a situation where there is an almost infinite number of paths.
[0091] In some cases, when a flow is remapped, an embodiment avoids storing a record of a previous path that a particular flow used to traverse the network. It is often the case that while a path may be congested at one point in time, the path may be open and no longer congested at a later point in time. Therefore, there is no significant benefit in avoiding the use of a previous path followed by a flow. For example, it is often the case that a flow initially follows path "A". Afterwards, the flow may be remapped to path "B", and then may be remapped to path "C". At some later point in time, the flow may be remapped back to path "A", and there may be no congestion on that path. In this sense, an embodiment can avoid or prevent the storage of data related to previous paths, because networks often fluctuate and just because a path was previously congested may not mean that the path is subsequently congested.
[0092] Thus, the disclosed embodiments enable endpoints to monitor network flow metrics (e.g., round-trip delay, effective throughput, number of ECN-marked or congestion notification packets, etc.) over time to detect and remap flows that are mapped to oversubscribed links. Remapping of flows is accomplished by causing the sending endpoint to change a field (e.g., source port value) in the packet header that is used as input to the switch's ECMP hash function. A remapping algorithm (e.g., a remapping probability function), which may be centralized or distributed, collects per-flow metrics within an observation window and may be used to determine whether and which flow to remap on each mapping iteration.
[0093] Compared to traditional ECMP, which is limited to random path selection caused by the hashing function of switches, the disclosed technology can adapt to changing network conditions and dynamically load balance flows to significantly or even completely eliminate congestion caused by link oversubscription. The disclosed technology is lightweight and does not require any network architecture changes because all remapping logic can be implemented in network endpoints.
[0094] Example Computer / Computer System
[0095] Now will focus on Fig.10 , Fig.10 An example computer system 1000 is illustrated, which may include and / or be used to perform any of the operations described herein. In addition, the computer system 1000 may be any of the endpoints or switches mentioned herein.
[0096] Computer system 1000 can take a variety of different forms. For example, computer system 1000 can be embodied as a tablet computer 1000A, a desktop or laptop computer 1000B, a wearable device 1000C, a mobile device, or any standalone device represented by ellipsis 1000D. Computer system 1000 can also be a distributed system that includes one or more connected computing components / devices that communicate with computer system 1000.
[0097] In its most basic configuration, computer system 1000 includes a variety of different components. Fig.10 Computer system 1000 is shown including one or more processors 1005 (also referred to as “hardware processing units”) and storage device 1010 .
[0098] With respect to processor(s) 1005, it should be understood that the functionality described herein may be performed, at least in part, by one or more hardware logic components (e.g., processor(s) 1005). For example, but not limitation, illustrative types of hardware logic components / processors that may be used include field programmable gate arrays (“FPGAs”), application specific or application specific integrated circuits (“ASICs”), application specific standard products (“ASSPs”), system on a chip (“SOCs”), complex programmable logic devices (“CPLDs”), central processing units (“CPUs”), graphics processing units (“GPUs”), or any other type of programmable hardware.
[0099] As used herein, the terms "executable module," "executable component," "component," "module," or "engine" may refer to a hardware processing unit or a software object, routine, or method that may be executed on the computer system 1000. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on the computer system 1000 (e.g., as separate threads).
[0100] Storage device 1010 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage devices, such as physical storage media. If computer system 1000 is distributed, processing, memory, and / or storage capabilities may also be distributed.
[0101] Storage device 1010 is shown as including executable instructions 1015. Executable instructions 1015 represent instructions executable by processor(s) 1005 of computer system 1000 to perform disclosed operations (eg, operations described in various methods).
[0102] The disclosed embodiments may include or utilize a special-purpose or general-purpose computer including computer hardware as discussed in more detail below, such as, for example, one or more processors (such as (multiple) processors 1005) and system memory (such as storage device 1010). Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are "physical computer storage media" or "hardware storage devices." In addition, computer-readable storage media including physical computer storage media and hardware storage devices exclude signals, carriers, and propagation signals. On the other hand, computer-readable media that carry computer-executable instructions are "transmission media" and include signals, carriers, and propagation signals. Therefore, by way of example and not limitation, the current embodiments may include at least two distinct types of computer-readable media: computer storage media and transmission media.
[0103] Computer storage media (also referred to as "hardware storage devices") are computer-readable hardware storage devices such as RAM, ROM, EEPROM, CD-ROM, RAM-based solid-state drives ("SSD"), flash memory, phase-change memory ("PCM") or other types of memory, or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store desired program code components in the form of computer-executable instructions, data or data structures and that can be accessed by a general or special purpose computer.
[0104] The computer system 1000 may also be connected (via a wired or wireless connection) to external sensors (e.g., one or more remote cameras) or devices via the network 1020. For example, the computer system 1000 may communicate with any number of devices or cloud services to obtain or process data. In some cases, the network 1020 itself may be a cloud network. In addition, the computer system 1000 may also be connected to a remote / separate (multiple) computer system via one or more wired or wireless networks, and the remote / separate (multiple) computer system is configured to perform any processing described with respect to the computer system 1000.
[0105] A "network" similar to network 1020 is defined as one or more data links and / or data switches that enable electronic data to be transmitted between computer systems, modules and / or other electronic devices. When information is transmitted or provided to a computer through a network (hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately regards the connection as a transmission medium. Computer system 1000 will include one or more communication channels for communicating with network 1020. Transmission media include networks that can be used to carry data or desired program code components in the form of computer executable instructions or in the form of data structures. In addition, these computer executable instructions can be accessed by general or special-purpose computers. The above combination should also be included in the scope of computer-readable media.
[0106] Upon reaching various computer system components, program code components in the form of computer executable instructions or data structures may be automatically transferred from transmission media to computer storage media (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be cached in RAM within a network interface module (e.g., a network interface card or "NIC"), and then ultimately transferred to computer system RAM and / or a less volatile computer storage medium at the computer system. Thus, it should be understood that computer storage media may be included in computer system components that also (or even primarily) utilize transmission media.
[0107] Computer executable (or computer interpretable) instructions include, for example, instructions that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Computer executable instructions can be, for example, binary code, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described using a language dedicated to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the above-mentioned features or actions. Instead, the described features and actions are disclosed as example forms of implementing the claims.
[0108] Those skilled in the art will appreciate that the embodiments can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, etc. The embodiments can also be practiced in a distributed system environment, in which local and remote computer systems that are linked (by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network each perform a task (e.g., cloud computing, cloud services, etc.). In a distributed system environment, program modules can be located in both local and remote memory storage devices.
[0109] The present invention may be embodied in other specific forms without departing from the characteristics of the present invention. The described embodiments are considered to be illustrative and non-restrictive in all respects. Therefore, the scope of the present invention is indicated by the appended claims rather than by the preceding description. All changes within the meaning and scope of the equivalents of the claims are included within their scope.
Claims
1. A method for triggering a flow to follow a new path through a network, wherein the flow comprises a plurality of network packets having similar characteristics, so that switches in the network route the plurality of network packets through the same path, wherein, By triggering the flow to follow the new path, congestion in the network is attempted to be reduced, the method comprising: identifying a plurality of flows traversing the network; analyzing network feedback data to determine that the plurality of flows conflict with each other and congest one or more switches of the network; Selecting a specific flow included in the plurality of flows to be remapped so that the specific flow will be converted from following a first path through the network to following a second path through the network, wherein remapping the specific flow includes, for each new network packet in the specific flow, performing the following operations: selecting a field within a header of each new network packet; Modify the value in the selected field; After modifying the value, performing hashing on the field value in the header including the modified value to generate a hash value for each new network packet; and indexing the hash value into an available path for each new network packet, wherein the available path is the second path; and New network packets of the particular flow are caused to follow the remapped second path through the network.
2. The method according to claim 1, wherein: The network feedback data includes one or more of: round trip delay for a network packet, an explicit congestion notification (ECN) marking on a network packet, an effective throughput metric, a buffer status, or a congestion notification packet (CNP) metric.
3. The method according to claim 1, wherein: Selecting the particular flow is based on a remapping probability function that determines which flow of the plurality of flows to remap based on relative congestion levels for the plurality of flows.
4. The method according to claim 1, wherein: The method is iterative until a convergence threshold is met.
5. The method according to claim 1, wherein: The field selected includes a source port value.
6. The method according to claim 1, wherein: Performing the hashing on the field values in the header includes performing the hashing on values in: an Internet Protocol (IP) field, an ingress interface field, a source address field, and a destination address field.
7. The method according to claim 6, wherein: The source address field includes a source port value.
8. The method according to claim 1, wherein: Modifying the value in the selected field includes modifying a particular value, the modification of which will not result in a change with respect to the location to which the particular stream is delivered.
9. The method according to claim 1, wherein: The second path differs from the first path in that the second path includes at least one switch that is not included in the first path.
10. The method according to claim 1, wherein: The first path and the second path are both available for routing the particular flow from a source endpoint to a destination endpoint, and wherein a third path is also available for routing the particular flow from the source endpoint to the destination endpoint.
11. The method according to claim 1, wherein: When the particular flow is remapped, storing a record of the previous path taken by the particular flow to traverse the network is avoided.
12. The method according to claim 1, wherein: The network feedback data is received from switches in the network.
13. The method according to claim 1, wherein: The endpoint is one of a plurality of endpoints in the network, and wherein global visibility is not provided for the plurality of endpoints.
14. The method according to claim 1, wherein: After the value in the selected field is modified, performing the hash on the field value in the header causes random generation of the second path such that the second path is not a pre-selected path.
15. The method according to claim 1, wherein: After the new network packets for the particular flow are caused to follow the remapped second path, new network feedback data is received and used to determine whether the network is still congested.
