Expanding flow channel across structure

Abstract

The disclosure relates to expanding flow channels across structures. One aspect of the present application provides a system and method for extending a flow from a first network fabric into a second network fabric. During operation, the system may identify, at an egress edge node of the first network fabric, a flow to which a received packet belongs based on a first flow identifier associated with the packet. The egress edge node may forward the received packet to a second network fabric, where a corresponding node in the second network fabric may identify the flow based on a second flow identifier. The system may indicate to an upstream node that forwarded packets exit the first network fabric and maintain the flow activity within the first network fabric after all packets in the flow exit the first network fabric and until all packets reach a destination node.

Description

[0001] Statement of Government Support

[0002] This invention was made with government support under contract number H98230-15-D-0022 / 0003 awarded by the Maryland Procurement Office. The government has certain rights in the invention. BACKGROUND TECHNICAL FIELD

[0003] The present disclosure relates to implementing flow-based congestion control in a network. More specifically, the present disclosure relates to the identification of flows (e.g., using connected flow lanes) across multiple fabrics. BACKGROUND

[0004] A variety of mechanisms have been developed to manage congestion in computer networks. Some of these mechanisms are around the concept of a "flow," where a flow can include a series of related packets that all have the same source and destination endpoints, and can also have other similar characteristics, such as belonging to the same traffic class, or being part of a single communication (e.g., as in the case of a TCP flow). Separating packets into different flows of traffic and then managing the progress of these flows independently can significantly reduce congestion within the overall system, as individual flows that are not making good progress can be slowed down without simultaneously slowing down all other flows. Thus, when all different flows of traffic are individually controlled, widespread congestion caused by a single flow not making good progress does not occur in systems without traffic separation capabilities. One mechanism that can operate at the link level and is capable of separating all packets into separate individual flows is known as a "flow lane."

[0005] Flow lanes serve as a fundamental mechanism in network traffic management. They can effectively separate data packets destined for different endpoints while ensuring that packets bound for the same destination remain grouped together and ordered in sequence. Furthermore, the implementation of flow lanes enables comprehensive monitoring of key network parameters. This includes tracking the precise path of packet flows, quantifying the amount of data transmitted, logging acknowledgments of successful packet delivery, and identifying instances of congestion throughout the network. By providing this detailed real-time insight into network behavior, flow lanes can significantly improve the speed and efficacy of congestion control measures.

[0006] Embodiments using current flow identification of a flow tunnel are typically limited to a single switching fabric where all devices operate under a uniform set of management policies. Within this framework, a flow can be initiated at an ingress point of the fabric, traverse a specific path through the fabric, and terminate at an egress point. However, this approach faces significant challenges in a multi-fabric network environment where each fabric can be managed by different administrative domains and operational practices. This limitation hinders the ability to implement seamless end-to-end congestion management in more complex multi-fabric network architectures, necessitating innovative solutions to extend flow tunnels beyond the boundaries of a single switching fabric. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 An example network environment is illustrated in accordance with an aspect of the present application.

[0008] Figure 2 An architecture of an example network node is illustrated in accordance with an aspect of the present application.

[0009] Figure 3 A flow diagram illustrating an example process for extending a flow tunnel from an upstream fabric to a downstream fabric is presented in accordance with an aspect of the present application.

[0010] Figure 4 An example functional block diagram of a network device is illustrated in accordance with an aspect of the present application.

[0011] Figure 5 A computer readable medium facilitating separation of flows is illustrated in accordance with an aspect of the present application.

[0012] In these drawings, like reference numbers refer to like elements throughout. DETAILED DESCRIPTION

[0013] According to some aspects of the present application, data packets injected into a network can be classified into different packet flows (or simply flows) based on a hash computed through their header fields, including destination, traffic class, and other appropriate fields. A set of flow tunnels not only encompasses the physical path traversed by these flows, but also the associated configuration information maintained by network devices (such as switches) along this route. Flow-based congestion control allows each node (e.g., a switch or router) along the data path to monitor and manage the congestion level of individual flows, enabling fast and effective congestion control and allowing the network to operate at higher capacity.

[0014] Traditional implementations of flow channels are confined within a single fabric, making end-to-end congestion management based on flow channels across multiple independently managed network applications a challenge. The present disclosure provides a mechanism for extending flow channels (i.e., the ability to track flows) across multiple independently managed fabrics. To keep the flow alive, the flow scope (a parameter that tracks the total length of pending packets within a flow) is set to a non-zero small value at the egress port of the source fabric when the flow leaves the source fabric and enters a downstream fabric.

[0015] In a fabric implementing flow-based congestion control, each flow channel can be marked by a unique identifier (e.g., a flow ID). For example, an ingress switch of the fabric can assign a flow ID to packets belonging to the same flow. The flow ID can be a locally valid value specific to a link, and the value can be unique only for a particular input port on a node. When a packet is forwarded to a next-hop node, the packet enters another link, and a new flow ID can be selected for the next flow channel of the next link. More specifically, each link can have its own set of flow channels in each direction, identified by their respective flow IDs. As the packets of a flow traverse multiple links and nodes, the flow IDs corresponding to the flow can form a unique chain of flow channels. At each node, the flow ID of an incoming packet can be used to map an entry in an input flow channel table (IFCT), which stores state information for the corresponding flow. An outgoing packet can be updated to a flow ID used by an outgoing link, and a mapping between the incoming flow ID and the outgoing flow ID can be stored in an output flow channel table (OFCT). This one-to-one mapping between flow IDs from upstream to downstream can start at an ingress edge node and end at an egress edge node. Because the flow ID only needs to be unique within an incoming link, a node can accommodate a large number of flows.

[0016] Flow channels can be established and released dynamically or "on the fly" based on demand. Specifically, when an initial packet of a flow arrives and no flow ID has been previously assigned to the flow, a flow channel is established at the ingress node (e.g., a mapping of flow ID to packet header is established). As this initial packet travels through the network, a flow ID can be assigned at each node along the path that the packet traverses, and a chain of flow IDs (i.e., sequentially connected flow channels) is established from the ingress node to the egress node. Subsequent packets belonging to the same flow use the same chain of flow IDs along the data path. When the packet is delivered to the destination egress node, the egress node can generate an acknowledgement (ACK) packet and send the acknowledgement packet in the upstream direction along the same data path to the ingress node. The ACK packet can indicate the amount of data that has been acknowledged. Upon receiving the ACK packet, each node along the data path can update its state information with respect to the amount of uncompleted and unacknowledged data for the flow. More specifically, the amount of data that has been transmitted but not acknowledged can be indicated by a variable called "flow-extent." At each node, the flow-extent value can be incremented for each transmitted packet and decremented for each received ACK packet. When the flow-extent at a node reaches zero, meaning that there is no more unacknowledged data, the node can release the flow ID (i.e., release that segment of the flow channel) and reuse the flow ID or channel for other flows. When the ACK for the last packet in the flow traverses the network, it can decrease the flow-extent value to zero at each node, thereby releasing the corresponding segment of the flow channel along the data path in reverse order. When the ACK for the last packet in the flow arrives at the ingress node, the entire flow is released or torn down.

[0017] In traditional approaches, flow channels are implemented within a single fabric, and flow IDs can be mapped to the fabric destination address of a packet. When a packet is received, an address translation is performed to translate the external media access control (MAC) or Internet Protocol (IP) address in the packet header to an internal fabric address. In cases where multiple independently managed system deployments are at a single site (e.g., supercomputer systems and storage systems are deployed at a weather forecasting site), each system can have its own fabric and header translation requirements, which means that ingress nodes in an ingress fabric are unaware of the fabric addresses of egress nodes in an egress fabric. To extend flow channels across multiple fabrics, according to some aspects of the present application, packets can be separated into different flow channels based on a single large hash value that is computed using multiple header fields in the packet without the need for header translation. The header fields can include, but are not limited to: Internet Protocol (IP) address fields (e.g., source / destination addresses), User Datagram Protocol (UDP) port fields (e.g., source / destination ports), traffic class fields, Differentiated Services Code Point (DSCP) fields, flow label fields, Virtual Network Identifier (VNI) fields, job identifier fields, UEC entropy fields, snooping number fields, etc. Additional examples of header fields can include Ethernet Layer 2 (L2) headers, Internet Protocol (IP) version 4 (IPv4) or IPv6 Layer 3 (L3) headers, and / or Layer 4 (L4) headers such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) headers. If the packet has been encapsulated for network overlay or other purposes, the L2, L3, and / or L4 headers of the encapsulated packet can also be included. Any fields extracted by the packet parser from multiple headers of layered protocols can be included in the hash computation. Additional header information, including but not limited to source port and other metadata that can be included in subsequent translation lookups, can also help generate the hash value. Entropy values obtained from local storage (e.g., control and state registers) can also be included.

[0018] In some aspects, the hash value is computed based on all of the header fields of an incoming packet to ensure a large enough entropy so that flow separation will be adequate no matter how many fabrics the flow traverses. However, packet header fields that can change for a given flow (e.g., Explicit Congestion Notification (ECN) fields used to indicate congestion or packet sequence numbers) should preferably not be included in the hash computation because these fields can cause packets belonging to a single flow to be separated into multiple flows. Separating a flow into multiple flows can cause packets to be out of order, which is generally undesirable.

[0019] Cross-structure extended flow identification allows for end-to-end congestion management, where congestion detected in a downstream structure can be reported to nodes in an upstream structure. In some examples, end-point or intermediate-structure congestion in a downstream structure can be considered when injecting packets into an upstream structure. Additionally, the structures need to be decoupled so that data load in a downstream structure does not affect an upstream structure, and vice versa. Because the structures are independently managed and can have different injection limits (i.e., the amount of data that can be injected into a structure at its ingress), it is desirable to ensure that load in a downstream structure does not significantly affect traffic injection into an upstream structure. Injection limits can prevent more data from being injected than is needed to maintain a desired bandwidth.

[0020] In the case where a node in an upstream structure can not receive an acknowledgement of a packet due to an error in a downstream structure, a timeout mechanism can be used to terminate the flow after a predetermined interval without receiving an ACK, even if the flow has left the structure.

[0021] One method of decoupling an upstream structure and a downstream structure is to configure the egress node of the upstream structure to acknowledge a packet as soon as the packet leaves the structure. However, sending an ACK packet upstream will cause the flow scope value on each node along the data path to decrease, and when the flow scope value on the egress node in the upstream structure decreases to zero, the flow channel is released even though packets belonging to the same flow are still propagating in the downstream structure. If the flow channel is no longer active, congestion information associated with the flow in the downstream structure will not be reported to nodes in the upstream structure, making it impossible to implement end-to-end congestion management.

[0022] To keep the flow channel active in the upstream fabric, according to some aspects of the application, the egress node of the upstream fabric can be configured to acknowledge the packet as it leaves the fabric, but set the flow range value to a non-zero small value. In one example, when the initial packet of the flow leaves the egress node, instead of generating and sending an ACK to acknowledge all the data in that initial packet, the egress node can generate and send an ACK that acknowledges all the data except for a small portion of the data. In other words, the edge node can suppress the acknowledgement of a small portion of the data in the initial packet, even after that data has been transmitted to the downstream fabric. The egress node can normally acknowledge the subsequent packets in the flow. The unacknowledged data in the initial packet can cause the flow range value at each node along the data path to not decrease to zero, thus keeping the flow open. Only after all the packets in the flow have reached their final destination egress node is the small portion of data acknowledged, and at that time the flow channel will be released. In some examples, the amount of flow range and data acknowledged by the ACK packet can be measured in a certain number of fixed length data units. In one example, one data unit (also referred to as a flow unit) can include 256 bytes. In another example, the flow range value for the flow at the egress node of the upstream fabric can be set to one data unit / flow unit after all the packets in the flow have left the fabric.

[0023] Setting the flow range to a small non-zero value (e.g., one flow unit) at the boundary of the fabric not only ensures that the flow channel remains active until all the packets reach their final destination, but also decouples the fabric, such that the data load in the downstream fabric does not affect the upstream fabric, and vice versa. Furthermore, because the flow channel remains open as the packets propagate in the downstream fabric, the congestion status regarding the flow channel can be updated by other ACK packets (e.g., congestion ACKs), thus facilitating end-to-end flow control.

[0024] Figure 1 An example network environment according to one aspect of the application is illustrated. In Figure 1 The network environment 100 can include two independently managed systems. The first system can include a server 102 and a switch fabric 104, and the second system can include a server 112 and a switch fabric 114. Each switch fabric can include multiple interconnected switches. For example, the switch fabric 104 includes switches 106 and 108, and the switch fabric 114 includes switches 116 and 118.

[0025] Figure 1A path of flow 120 is also shown, which is established between the ingress port to which server 102 is connected and the egress port to which server 112 is connected, as shown by the dashed line. As previously discussed, when an initial packet belonging to flow 120 is injected into ingress switch 106 of fabric 104, ingress switch 106 can assign a flow ID (i.e., allocate a new flow channel) to the packet that is unique to the input port that receives the packet. To ensure that the flow channel can be extended from fabric 104 to fabric 114, the flow ID can be mapped to a large hash value that is generated based on multiple unaltered header fields of the injected packet. By computing the large hash value and mapping it to a locally unique flow ID, packets belonging to different flows can be separated into different flow channels at the ingress.

[0026] The ingress nodes maintain an edge flow channel table (EFCT) that stores mappings between large hash values and flow IDs. In addition, each node maintains an IFCT that stores state information associated with each flow, such as a flow range. An entry in the IFCT corresponding to a flow can also include a flow-specific injection limit that can control the amount of data injected into the fabric by the flow. To facilitate flow-based congestion control, the injection limit can be compared to the flow range value in the IFCT of each node that the flow traverses. If the flow range value is greater than the injection limit at a node (e.g., more data has been injected than allowed), the node can stop forwarding packets belonging to the flow. In addition to per-flow injection limits, the ingress nodes can maintain per-traffic class injection limits that limit the amount of data injected into the fabric by all flows in a given traffic class. In this case, the sum of the flow range values of all active flows in a traffic class is compared to the traffic class-specific injection limit. If the sum of the flow range values for a particular traffic class is greater than the corresponding injection limit, packets belonging to the traffic class can be suspended from being injected into the fabric.

[0027] According to some aspects, injection limits (e.g., per-flow or per-traffic class limits) can be set independently at the ingress nodes of each fabric. For example, injection limits can be set at ingress node 106 of fabric 104 and at ingress node 116 of fabric 114. If the ingress nodes include multiple ingress ports, the injection limits can be set on a per-ingress port basis. More specifically, the system can measure the amount of data that has been injected into the fabric on a per-ingress port basis and set the injection limit to cap the total amount of data that the port can inject into the fabric.

[0028] At the egress node 108 of the upstream fabric 104, the flow range of the flow can be set to a non-zero small value (e.g., one flow unit) by immediately acknowledging all but a small portion of the data leaving the fabric 104. In some examples, all but the initial packet in the flow can be immediately acknowledged in their entirety as they leave the egress node 108, thereby reducing the flow range value in the upstream nodes. In one example, the initial packet of the flow can comprise 10 flow units (e.g., 2560 bytes). In response to receiving the initial packet, the egress node 108 can generate and send back an ACK packet acknowledging 9 flow units, leaving one flow unit of data in the initial packet unacknowledged. As the ACK packet traverses the reverse data path toward the ingress node 106, the flow range value at each node can be set to at least one flow unit, thereby ensuring that the flow 120 remains active in the fabric 104. Note that the flow range value can be reset to its normal value at the egress port of the egress node 108 (which indicates the amount of data that has been transmitted and unacknowledged in the downstream fabric 114). At the ingress node 116 of the fabric 114, the flow range value of the flow is the normal value, and can be compared to the injection limit at the ingress node 116 to determine whether to forward a packet in the flow.

[0029] At the destination egress node (i.e., the egress node 118 of the downstream fabric 114), in response to receiving all of the packets in the flow, a flow termination ACK packet can be generated and sent back upstream. Upon receiving the flow termination ACK packet, the egress node 108 of the upstream fabric 104 can generate and send back to the ingress node 106 an ACK packet for the small portion of unacknowledged data to terminate the flow 120 in the fabric 104.

[0030] According to alternative aspects, other mechanisms can be used to keep a flow open in the upstream fabric. In some examples, the ACK packet generated and returned by the destination edge node can include a keep tunnel open flag field that is used to indicate to all upstream nodes that the flow tunnel should be kept open. The keep tunnel open flag field in the last ACK packet for a flow can be reset to indicate that the flow tunnel can be released.

[0031] Figure 1The illustrated example includes two coupled fabrics. In practice, the same solution (including acknowledging data packets as they leave the fabric and keeping the flow open until all packets reach their final destination) can be extended to more than two fabrics (e.g., three or more coupled fabrics). In one example, the egress edge node of each fabric can be configured to acknowledge all transmitted data except for a small fraction to keep the flow tunnel active. In another example, the egress edge node of each fabric can generate and send an ACK packet with a keep tunnel open flag field to inform upstream nodes to keep the flow tunnel active.

[0032] Figure 1 Each node in the network 100 is a computing device, which can be any single computing device, a group of computing devices, a portion of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. According to some aspects, a computing device is any device, portion of a device, or any collection of devices capable of electronically processing instructions, and can include, without limitation, any one of the following: one or more processors (e.g., components including circuitry) (not shown), memory (e.g., random access memory (RAM)) (not shown), input and output devices(s) (not shown), non-volatile storage hardware (e.g., solid state drive (SSD), persistent memory (Pmem) device, hard disk drive (HDD) (not shown)), one or more physical interfaces (e.g., network ports, storage ports) (not shown), any number of other hardware components (not shown), and / or any combination thereof.

[0033] Examples of computing devices include, without limitation, servers (e.g., blade servers in a blade server chassis, rack servers in a rack, etc.), desktop computers, mobile devices (e.g., laptop computers, smartphones, personal digital assistants, tablet computers, automotive computing systems, and / or any other mobile computing devices), storage devices (e.g., arrays of disk drives, fiber channel storage devices, Internet Small Computer System Interface (iSCSI) storage devices, tape storage devices, arrays of flash storage devices, network attached storage devices, etc.), network devices (e.g., switches, routers, multi-layer switches, etc.), virtual machines, virtualized computing environments, logical containers (e.g., for one or more application programs), Internet of Things (IoT) devices, arrays of nodes of computing resources, supercomputing devices, data centers or any portion thereof, and / or any other type of computing device having the aforementioned requirements.

[0034] Figure 2 An architecture of an example network node according to an aspect of the present application is illustrated. In Figure 2In particular embodiments, network node 200 can include ingress port 202, flow identification function 204, EFCT 206, data crossbar 208, egress port 210, OFCT 212, ACK generation function 214, ACK crossbar 216, flow maintenance function 218, and IFCT 220. In some examples, network node 200 can be an egress edge switch of a switch fabric. The various components in network node 200 can be implemented using any form of hardware, software, or a combination thereof.

[0035] Ingress port 202 is responsible for receiving data packets from end hosts coupled to network node 200. Depending on the implemented communication protocol, data packets can include various headers. In one example, data packets can be Ethernet frames with Ethernet headers. In some examples, data packets can include a fabric header that includes a flow ID. According to some aspects, the flow ID can be mapped to a hash value computed using a plurality of untransformed header fields in the packet. The hash value can be computed based on header information associated with a plurality of fabrics as the packet traverses the plurality of fabrics. Examples of the plurality of header fields can include, but are not limited to, IP address fields (e.g., source / destination address), user datagram protocol (UDP) port fields (e.g., source / destination port), traffic class fields, DSCP fields, flow label fields, VNI fields, job identifier fields, UEC entropy fields, snooping number fields, etc. Additional examples of header fields can include L2 headers, IPv4 or IPv6 L3 headers, and / or L4 headers such as TCP or UDP headers. If the packet has been encapsulated for network overlay or other purposes, L2, L3, and / or L4 headers of the encapsulated packet can also be included. Any fields extracted by the packet parser, obtained from a plurality of headers of layered protocols, can be included in the hash computation. Additional header information, including but not limited to source port and other metadata that can be included in subsequent transform lookups, can also help generate the hash value. Entropy values obtained from local storage (e.g., control and state registers) can also be included.

[0036] The flow identification function 204 is responsible for identifying the flow to which a received packet belongs based on the flow ID included in the fabric header. According to some aspects, the flow identification function 204 can perform a lookup operation in the EFCT 206 based on a hash to identify an entry that matches a previously assigned flow ID. In some aspects, the EFCT 206 can be stored in a ternary content addressable memory (TCAM), such as a TCAM or any other hash-based lookup function suitable for an exact match operation. Any function capable of performing a match operation can also be implemented (i.e., any match function), such as an exact match hash function implemented using multiple RAMs, or a match function implemented using multiple discrete logic gates. If no matching entry is found in the EFCT 206, the packet belongs to a new flow and a new flow ID can be assigned for the flow at the input port of the network node 200. The flow ID can also be used to identify or allocate a flow-specific input queue (not shown in FIG. 2) in which the incoming packet can be temporarily stored. State information (e.g., flow state, congestion state, and forwarding state) and congestion information about the flow can be stored in the IFCT 220. The IFCT 220 can also store parameters used to monitor and control the flow-specific input queue. Figure 2

[0037] The data crossbar 212 is responsible for forwarding data packets from the flow-specific input queue to the egress port 210. The egress port 210 is responsible for sending outgoing packets to the next hop node. The egress port 210 can perform a lookup in the OFCT 212 using the flow ID included in the packet header. The lookup can return an outgoing flow ID, which can be used to update the flow ID in the packet header. The OFCT 212 can store information that can be used to calculate a flow range value for each active flow. For example, an entry in the OFCT 212 can include a data_flow field that tracks the amount of data that has been transmitted and an ACK_flow field that tracks the amount of data that has been acknowledged. The flow range value can be calculated based on the difference between the data_flow field and the ACK_flow field. In some examples, the OFCT 212 can store the flow range value for each active flow.

[0038] The ACK generation function 214 is responsible for generating ACK packets corresponding to packets leaving the switch fabric. More specifically, the ACK packets can indicate to an upstream node of the network node 200 that the corresponding packet is leaving the switch fabric, thereby reducing the flow range and allowing new packet data equal in size to the data represented in the ACK packet to pass the injection limit and be allowed into the fabric. In other words, when a per-flow injection limit is set at the ingress of the switch fabric, the packets that have left are not considered part of the load in the switch fabric. Timely acknowledgement of transmitted data packets can effectively decouple the load in the downstream fabric from the upstream fabric. The generated ACK packets can be forwarded upstream via the ACK crossbar 216.​

[0039] The network node 200 further includes a flow maintenance function 218 configured to keep the flow channel active within the switch fabric after all packets in the flow channel have left the switch fabric and until all packets reach the destination node. According to some aspects, the flow maintenance function 218 can configure the ACK generation function 214 to acknowledge all transmitted data except for a small fraction, thereby preventing the flow scope value from decreasing to zero after all packets in the flow have left the switch fabric. According to alternative aspects, the flow maintenance function 218 can include logic that can reset the flow scope value directly to a fixed small value (e.g., one or a few flow units) stored in the OFCT 212 after all packets in the flow have left the switch fabric. Maintaining the flow channel across the fabric boundary ensures that congestion in the downstream fabric (e.g., endpoint congestion and intermediate fabric congestion) can be reported (via ACK packets) to the nodes in the upstream fabric. Thus, when setting the injection limit at the ingress of the upstream fabric, the congestion in the downstream fabric will be taken into account.

[0040] Further, extending the flow channel across multiple fabrics enables the congestion management system to distinguish between endpoint or intermediate fabric congestion in the downstream fabric and congestion on a link aggregation group (LAG) between the upstream fabric and the downstream fabric. Extending the flow channel across multiple fabrics also allows the entire flow to be rerouted to a different link, thus connecting the two fabrics together without reordering any packets within the flow. All packets of the flow maintain their original order. More specifically, congestion in the downstream fabric is reported via ACK packets (which can be generated at the destination edge switch or intermediate switches), while congestion in the LAG can be reflected by the queue state. Congestion in the LAG can be mitigated via various load balancing techniques. In some examples, the ACK generation function 214 can generate a redirect ACK that can be used to redirect traffic among different links in the LAG.

[0041] In addition to setting the flow scope value in the OFCT 212 to a small value, according to some aspects, the flow maintenance function 218 can also set a keep-channel-open flag field in the ACK packets generated by the ACK generation function 214, thereby indicating to all upstream nodes that the corresponding flow channel should be kept open.

[0042] Flow maintenance function 218 is also responsible for deactivating the flow channel after all packets in the flow have reached their final destination. For example, after determining that all packets in the flow have reached their final destination (e.g., after receiving an ACK for the last packet in the flow from the destination edge switch), flow maintenance function 218 can configure ACK generation function 214 to generate ACK packets to acknowledge a small portion of previously unacknowledged data in the initial packet, thereby reducing the flow range value in network node 200 to zero. In an alternative example, flow maintenance function 218 can update the keep-channel-open flag field in the ACK packet to indicate to upstream nodes that the flow channel is ready to be released.

[0043] Depending on some aspects, network node 200 may also include header modification functionality. Figure 2 (Not shown in the image), this header modification function is responsible for adding, removing, or modifying the header of data packets. This capability allows data packets to tunnel through third-party structures (e.g., by adding an encapsulation header or IP option field containing an identifier to be returned by the destination structure) while implementing end-to-end flow control. Furthermore, congestion information detected in third-party structures can be parsed at the edge of the upstream structure and used to update flow state information.

[0044] Figure 3 A flowchart illustrating an example process for extending a flow channel from an upstream structure to a downstream structure, according to one aspect of this application, is presented. Figure 3 All or any part of the operation shown can be performed, for example, by a device or a group of devices (e.g., Figure 1 and Figure 2 The execution is performed by either the export edge node 108 or the network node 200, as shown respectively. Although Figure 3 The example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart can be performed in different orders, and can be performed simultaneously, partially simultaneously, or in combination.

[0045] During operation, nodes in the upstream structure can receive data packets (operation 302). Nodes can be exit edge nodes that couple the upstream structure to the downstream structure (e.g., Figure 1 (Node 108 shown). Depending on the communication protocol implemented, data packets can be Transmission Control Protocol (TCP) packets, UDP datagrams, IP packets, Ethernet packets, etc.

[0046] The flow identification logic unit implemented on the node can identify the flow channel to which the received packet belongs based on a first flow identifier associated with the received packet (operation 304). The flow identification logic unit can be similar to... Figure 2The first flow ID can be included in a structure header of the received packet. The first flow ID can be mapped at an ingress node of the structure to a first hash value computed based on header information associated with the upstream structure. The first flow ID allows a flow lane to be uniquely identified in the first structure. Examples of packet header information can include, but are not limited to, IP address information (e.g., source / destination address), UDP port information (e.g., source / destination port), traffic class information, DSCP information, flow label information, packet encapsulation information (e.g., VNI), UEC entropy information, snooping metadata (e.g., snoop number), and the like.

[0047] An egress port of the node can forward the packet to a downstream structure (operation 306). In one example, the packet can be forwarded to an ingress node of the downstream structure via a LAG. A node in the downstream structure can identify the flow lane based on a second flow ID. The second flow ID can be mapped at an ingress node of the downstream structure to a second hash value computed based on a plurality of headers associated with the downstream structure. Examples of header fields can include, but are not limited to, a source address field, a destination address field, a traffic class field, an encapsulation header field, a differentiated services code point (DSCP) field, a user datagram protocol (UDP) port field, one or more ultra-ethernet consortium (UEC) transport headers, or a snoop number field. The second flow ID allows a flow lane to be uniquely identified in the second structure.

[0048] After forwarding the packet to the downstream structure, the egress node can indicate to the upstream node that the data packet has left the upstream structure (operation 308). According to some aspects, ACK generation logic implemented on the egress node can generate an ACK packet to acknowledge the transmission of the data packet. In some examples, the ACK packet can specify a quantity of flow units corresponding to the acknowledged data. Timely acknowledgement of the outgoing data packet can prevent the packet from impacting packet injection into the flow lane at the ingress of the upstream structure. More specifically, when a per-flow injection limit is set at the ingress node of the upstream structure, the system does not include the outgoing packet as part of the data load in the upstream structure. The egress node can update a flow range value stored in its OFCT based on the ACK packet. The flow range value tracks outstanding (i.e., transmitted but unacknowledged) data in the flow lane within the first network structure based on the ACK packet.

[0049] After all packets in the flow have exited the upstream fabric, the egress node keeps the flow channel active until all packets reach their final destination (operation 310). According to some aspects, the ACK generation logic can be configured to acknowledge all data in the flow except for a small amount of data (e.g., one or a few flow units) to prevent the flow extent from reducing to zero. The non-zero flow extent value ensures that the flow channel remains active. According to alternative aspects, the ACK generation logic can be configured to generate ACK packets with a keep channel open flag field to inform edge nodes and any upstream nodes receiving the ACK packets to keep the flow channel active.

[0050] After all packets in the flow have reached the destination node (e.g., transmitted by the destination edge switch in the downstream fabric to the destination server), the destination edge switch can generate and return a flow termination ACK packet upstream. Upon receiving the flow termination ACK packet, the egress edge node in the upstream fabric can acknowledge the last small amount of data in the flow to release or terminate the flow channel in the upstream fabric. According to alternative aspects, the flow termination ACK packet can have its keep channel open flag field unset to inform upstream nodes to terminate the flow channel.

[0051] Figure 4 An example functional block diagram of a network device according to one aspect of the present application is illustrated. Network device 400 can include any physical device that allows hardware on a computer network to communicate and interact with each other. Examples of network device 400 can include switches, routers, gateways, access points, network interface cards (NICs), etc. In Figure 4 In particular, network device 400 can include a plurality of communication ports, such as ports 402 and 404, for communicating with peer network devices. Each port can include a transmitter and a receiver.

[0052] Network device 400 can include one or more processing resources (e.g., processing resource 406), one or more storage devices (e.g., storage device 408), and a flow extension system 410. Network device 400 can include fewer or more entities than those shown in Figure 4 In particular, network device 400 can include a plurality of communication ports, such as ports 402 and 404, for communicating with peer network devices. Each port can include a transmitter and a receiver.

[0053] In examples described herein, a processing resource can include one processor or multiple processors, e.g., included in a single computing device or distributed across multiple computing devices. As used herein, a “processor” can be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to retrieve and execute instructions, other electronic circuitry suitable for retrieving and executing instructions stored on a computer-readable storage medium, or a combination thereof. In examples described herein, a processing resource can fetch, decode, and execute instructions stored on a storage medium to perform the functionality described with respect to the instructions stored on the computer-readable medium. In other examples, the functionality described with respect to any of the instructions described herein can be implemented in the form of electronic circuitry, in the form of executable instructions encoded on a computer-readable medium, or a combination thereof. A computer-readable storage medium can be located either in the computing device executing the instructions or remote from the computing device but accessible (e.g., via a computer network) to the computing device for execution. In examples illustrated herein, a node can be implemented by one computer-readable storage medium or multiple computer-readable storage media.

[0054] The flow expansion system 410 can include any number of software units, hardware units, and firmware units that work together to achieve the goal of expanding flow lanes across at least the first structure and the second structure. According to some aspects, the flow expansion system 410 can include instructions that, when executed by the processing resource 406, can cause the processing resource 406 to perform the methods and / or processes described in this disclosure. In particular, the flow expansion system 410 can include instructions for identifying a flow lane to which a packet received in the first structure belongs based on a first flow ID associated with the packet (as described above with respect to the method 300), expanding the flow lane across the first structure and the second structure (as described above with respect to the method 300), and / or any other functionality described in this disclosure. Figure 3Instruction 412 (described in operation 304) is shown. According to some aspects, the first flow ID can be determined based on a hash value calculated at the entry point of the first structure. The hash value can be calculated based on packet header information associated with the first network structure, thereby allowing unique identification of flow channels within the first network structure. Examples of packet header information may include, but are not limited to: IP address information (e.g., source / destination address), UDP port information (e.g., source / destination port), traffic category information, DSCP information, flow label information, packet encapsulation information (e.g., VNI), job identifier field, UEC entropy information, listening metadata (e.g., listening number), and so on. Additional examples of header fields may include L2 headers, IPv4 or IPv6 L3 headers, and / or L4 headers, such as TCP or UDP headers. If the packet has been encapsulated for network overlay or other purposes, L2, L3, and / or L4 headers encapsulating the packet may also be included. The hash calculation may include any fields extracted by the packet parser from multiple headers of the layered protocol. Additional header information (including, but not limited to, the source port and other metadata that may be included in subsequent transformation lookups) can also help generate the hash value. Entropy values ​​obtained from local storage devices (e.g., control and status registers) may also be included.

[0055] Stream extension system 410 may include a method for forwarding received packets to a second structure (as described above). Figure 3 Instruction 414 (described in operation 306 shown). According to some aspects, packets can be forwarded to the ingress node of the downstream structure via LAG. Nodes in the second network structure can identify flow channels based on a second flow identifier, which can be mapped to a hash value computed at the ingress of the downstream structure.

[0056] The flow extension system 410 may include a means for instructing upstream nodes to leave the first structure (as described above regarding...). Figure 3 The instruction 416 (described in operation 308) is shown in the diagram. According to some aspects, instruction 416 can be used to generate ACK packets to acknowledge the transmission of data packets. The ACK packet can specify the number of flow units corresponding to the acknowledged data. The egress node can update the flow range value stored in its OFCT based on the ACK packets.

[0057] The streaming extension system 410 may include features for maintaining streaming channel activity after all packets in the stream have left the first structure and until all packets have reached the destination node (as described above). Figure 3The instruction 418 (described in operation 310) is used to acknowledge all data in the stream except for a small portion of the data (e.g., one or a few stream units). According to some aspects, instruction 418 can be used to generate an ACK packet that acknowledges all data in the stream except for a small portion (e.g., one or a few stream units) to prevent the stream range from decreasing to zero. According to alternative aspects, instruction 418 can be used to generate an ACK packet with a keep-channel-open flag field to notify edge nodes and any upstream nodes receiving the ACK packet to keep the stream channel active.

[0058] The streaming extension system 410 may include more than Figure 4 The instructions shown are further examples of instructions. For instance, the flow extension system 410 may include instructions for terminating a flow channel in response to receiving a flow termination ACK packet from the destination node. The flow extension system 410 may also include instructions for independently setting injection limits at the ingress of upstream and downstream structures. The flow extension system 410 may further include instructions for performing flow-based congestion control. These instructions can be used to suspend packet injection into the corresponding network structure in response to the sum of the flow range values ​​of all active flow channels within the corresponding network structure exceeding the injection limit.

[0059] Figure 5 A computer-readable medium facilitating the separation of flows according to one aspect of this application is illustrated. CRM 500 can be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a "computer-readable storage medium" can be any electronic, magnetic, optical, or other physical storage device for containing or storing information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein can be any one or a combination of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., compact disk, DVD, etc.). Furthermore, any computer-readable storage medium described herein can be non-transitory.

[0060] CRM 500 can store: information used to identify the stream channel to which a packet belongs based on a first stream ID associated with the packet received in the first structure (as mentioned above). Figure 3 Instruction 510 (as described in operation 304) for forwarding the received packet to the second structure (as described above regarding...). Figure 3 Instruction 520 (as described in operation 306) is used to instruct the upstream node to leave the first structure (as mentioned above regarding...). Figure 3 Instruction 530 (as described in operation 308); and instruction 530 for maintaining the flow channel activity after all packets in the flow have left the first structure and until all packets have reached the destination node (as described above regarding...).Figure 3 instructions 540 (as described in operation 310).

[0061] CRM 500 can include more instructions than Figure 5 instructions than those shown in FIG. 5. For example, CRM 500 can include instructions for terminating a flow tunnel in response to receiving a flow termination ACK packet from a destination node. CRM 500 can also include instructions for independently setting an injection limit at an ingress of an upstream fabric and a downstream fabric. CRM 500 can further include instructions for performing flow-based congestion control. These instructions can be used to pause packet injection into a respective network fabric in response to a sum of flow extent values for all active flow tunnels within the respective network fabric being greater than the injection limit.

[0062] Generally, aspects of the present disclosure address the technical problem of extending a flow (i.e., the ability to track a flow) across multiple independently managed fabrics. When a packet in a flow leaves an upstream fabric and enters a downstream fabric, the packet is immediately acknowledged at an egress node of the upstream fabric to ensure that data load in the downstream fabric does not impact the upstream fabric, and vice versa. More specifically, injection limits of the two fabrics can be independently set. Furthermore, a flow can remain active until all packets in the flow reach their destination to facilitate end-to-end flow-based congestion control. In some examples, a flow extent (a parameter that tracks an amount of pending data within a flow) is set to a non-zero small value at an egress node of an upstream fabric to keep the flow active. At an ingress node of a downstream fabric, the flow extent is reset to its actual value to track pending data in the flow in the downstream fabric.

[0063] One aspect of the present application provides a system and method for extending a flow tunnel from a first network fabric into a second network fabric. During operation, the system can identify, at an egress edge node of the first network fabric, a flow to which a received packet belongs based on a first flow identifier associated with the received packet. The egress edge node can forward the received packet to the second network fabric, where a respective node in the second network fabric can identify the flow based on a second flow identifier. The system can indicate to an upstream node that the forwarded packet left the first network fabric, and keep the flow active within the first network fabric after all packets in the flow leave the first network fabric and until all packets reach a destination node.

[0064] In variations of this aspect, the first identifier or the second flow identifier corresponds to a hash value computed based on a plurality of header fields of the packet. The header fields can include one or more of: a source address field, a destination address field, a traffic class field, an encapsulation header field, a job identifier field, a differentiated services code point (DSCP) field, a user datagram protocol (UDP) port field, one or more ultra-ethernet consortium (UEC) transport headers, a listening number field, or any header field that uniquely identifies a flow in both the first network fabric and the second network fabric.

[0065] In variations of this aspect, indicating to the upstream node that the forwarded packet exited the first network fabric can include generating and sending an acknowledgement packet corresponding to the forwarded packet. The acknowledgement packet can specify an amount of acknowledged data.

[0066] In further variations, the system can update a flow-wide value used to track outstanding packets in the flow within the first network fabric based on the acknowledgement packet.

[0067] In further variations, maintaining the flow active includes suppressing acknowledgement of at least a portion of the initial packets in the flow to ensure that the flow-wide value is non-zero.

[0068] In further variations, the system can receive an acknowledgement of all packets in the flow from the destination node, generate and send an acknowledgement packet associated with the portion of the initial packets in the flow, and terminate the flow within the first network fabric.

[0069] In further variations, the system can set an injection limit at an ingress of the respective network fabric and perform flow-based congestion control in the respective network fabric, which can include pausing packet injection into the respective network fabric in response to a sum of flow-wide values of all active flows within the respective network fabric being greater than the injection limit.

[0070] In further variations, the system can perform flow-based congestion control further including receiving acknowledgements including downstream flow congestion information and using the information to control a maximum bandwidth of injected packets on each individual flow separately.

[0071] In further variations, the system can reset the flow-wide value at an egress of the first network fabric. The reset flow-wide value tracks outstanding packets in the second network fabric.

[0072] In further variations, the system can maintain the flow active by setting a keep- tunnel-open flag field in the acknowledgement packet.

[0073] One aspect of the present application provides a network edge node to couple a first network fabric and a second network fabric. The network edge node can include an ingress port to receive a packet from an upstream node within the first network fabric, a flow lane identification logic unit to identify a flow to which the received packet belongs based on a first flow identifier associated with the received packet, and an egress port to forward the packet to the second network fabric. A respective node in the second network fabric can identify the flow based on a second flow identifier. The network edge node can further include an indication logic unit to indicate to the upstream node that the forwarded packet leaves the first network fabric, and a flow maintenance logic unit to keep the flow active within the first network fabric after all packets in the flow leave the first network fabric and until all packets reach a destination node.

[0074] One aspect of the present application provides a non-transitory machine-readable storage medium storing instructions executable by a processing resource to identify, at an egress edge node of a first network fabric, a flow to which a received packet belongs based on a first flow identifier associated with the received packet, forward, by the egress edge node, the received packet to a second network fabric, a respective node in the second network fabric identifying the flow based on a second flow identifier, indicate to an upstream node that the forwarded packet leaves the first network fabric, and keep the flow active within the first network fabric after all packets in the flow leave the first network fabric and until all packets reach a destination node.

[0075] In this disclosure, functions include a number of logic units capable of performing predetermined logic functions described throughout this disclosure. Figure 2 and Figure 3 The functions shown in the flow diagrams of FIGS. 1-4 can be implemented using any form of hardware, software, or combinations thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routines or other mechanisms can be implemented to make up circuitry. In implementation, various functions described herein can be implemented as discrete circuits or the functions and features described can be shared among one or more circuits. Although various features or elements of functionality can be individually described or claimed as separate features or elements, these features and elements can be shared among one or more common functions and such description shall not require or imply that separate circuits or circuitry are required to implement such features or functionality.

[0076] The methods and processes described in the specific embodiments section can be embodied as code and / or data, which can be stored in a computer-readable storage medium as described above. When the computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer-readable storage medium.

[0077] The methods and processes described above can be included in a hardware module or apparatus. A hardware module or apparatus can include, but is not limited to, an application- specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA), a dedicated or shared processor that executes a particular software module or code, and other programmable logic devices now known or later developed. When activated, the hardware module or apparatus performs the methods and processes included within it.

[0078] The foregoing description of various aspects and examples has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit various aspects and examples to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of various aspects and examples described herein be limited not with this detailed description, but rather determined with reference to the claims as interpreted in accordance with the full breadth permitted by the patent laws.

[0079] Moreover, the previous description of the aspects is provided for illustrating and describing various aspects and examples. The descriptions are not intended to be exhaustive or to limit various aspects and examples to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the various aspects described herein. The scope of the various aspects described herein is defined with reference to the claims.

Claims

1. A method for extending the identifier of a flow from a first network structure to a second network structure, the method comprising: At the exit edge node of the first network structure, the flow to which the received packet belongs is identified based on a first flow identifier associated with the received packet; The received packets are forwarded to the second network structure by the egress edge node, and the corresponding node in the second network structure identifies the flow based on the second flow identifier; Instruct upstream nodes to allow forwarded packets to leave the first network structure; as well as The flow remains active within the first network structure after all packets in the flow have left the first network structure and until all packets have reached their destination node.

2. The method as described in claim 1, wherein, The first stream identifier or the second stream identifier corresponds to a hash value calculated based on a plurality of header fields of the packet, wherein the header fields include one or more of the following: Source address field; Destination address field; Traffic category field; Encapsulate header fields; Job identifier field; Differential Service Code Point (DSCP) field; User Datagram Protocol (UDP) port field; One or more Ultra Ethernet Alliance (UEC) headers are transmitted; Listen to the ID field; or Any header field that uniquely identifies the flow in both the first network structure and the second network structure.

3. The method as described in claim 1, wherein, Instructing the upstream node that a forwarded packet leaves the first network structure includes: generating and sending an acknowledgment packet corresponding to the forwarded packet, wherein the acknowledgment packet specifies the amount of data that has been acknowledged.

4. The method of claim 3, further comprising: The flow range value is updated based on the confirmed packets, and the flow range value is used to track pending packets in the flow within the first network structure.

5. The method of claim 4, wherein, Maintaining the flow activity includes: suppressing acknowledgments of at least a portion of the initial packets in the flow to ensure that the flow range value is non-zero.

6. The method of claim 5, further comprising: Receive acknowledgments for all packets in the stream from the destination node; Generate and send an acknowledgment packet associated with the portion of the initial packet in the stream; as well as The flow is terminated within the first network structure.

7. The method of claim 4, further comprising: Set injection restrictions at the entry point of the corresponding network structure; as well as Flow-based congestion control is performed in the corresponding network structure, the flow-based congestion control including: pausing packet injection in the corresponding network structure in response to the sum of the flow range values ​​of all active flows within the corresponding network structure being greater than the injection limit.

8. The method of claim 7, wherein, Performing the flow-based congestion control further includes: receiving an acknowledgment including downstream flow congestion information, and using the information to control the maximum bandwidth of packets injected on each individual flow, respectively.

9. The method of claim 4, further comprising resetting the flow range value at the exit of the first network structure, wherein, The reset flow range value is used to track pending packets in the second network structure.

10. The method of claim 3, wherein, Maintaining the flow activity includes setting a keep-channel-open flag field in the confirmation group.

11. A network edge node coupling a first network structure and a second network structure, the network edge node comprising: An ingress port, which is used to receive packets from upstream nodes within the first network structure; A flow identification logic unit, the flow identification logic unit being used to identify the flow to which a received packet belongs based on a first flow identifier associated with the received packet; An egress port is used to forward the packet to the second network structure, where a corresponding node in the second network structure identifies the flow based on a second flow identifier; An instruction logic unit is configured to instruct the upstream node that the forwarded packet has left the first network structure. as well as A flow maintenance logic unit is configured to maintain the flow activity within the first network structure after all packets in the flow have left the first network structure and until all packets have reached the destination node.

12. The network edge node as described in claim 11, wherein, The indication logic unit includes acknowledgment packet generation logic, which is used to generate and send an acknowledgment packet corresponding to the forwarded packet, wherein the acknowledgment packet specifies the amount of data that has been acknowledged.

13. The network edge node of claim 12, further comprising a flow range update logic unit, the flow range update logic unit being configured to update a flow range value based on the acknowledgment packet, the flow range value being configured to track pending packets in the flow within the first network structure.

14. The network edge node as described in claim 13, wherein, The flow maintenance logic unit is configured to suppress acknowledgments of at least a portion of the initial packets in the flow to ensure that the flow range value is non-zero.

15. The network edge node of claim 14, further comprising a flow termination logic unit, the flow termination logic unit being configured to: in, The acknowledgment packet generation circuit is configured to generate and send an acknowledgment packet in response to receiving acknowledgments for all packets in the stream from the destination node, to acknowledge the portion of the initial packet in the stream; and The flow termination logic unit is configured to terminate the flow within the first network structure in response to the acknowledgment of the portion of the initial packet.

16. The network edge node of claim 13, further comprising flow range reset logic, the flow range reset logic being configured to reset the flow range value at the egress port of the network node to track pending packets in the second structure.

17. The network edge node as described in claim 12, wherein, The stream sustaining logic unit is used to configure the acknowledgment packet generation logic to set a keep-channel-open flag field in the acknowledgment packet.

18. A non-transitory machine-readable storage medium storing instructions that can be executed by processing resources to perform the following operations: At the exit edge node of the first network structure, the flow to which the received packet belongs is identified based on the first flow identifier associated with the received packet; The received packets are forwarded by the egress edge node to the second network structure, where the corresponding nodes identify the flow based on the second flow identifier; Instruct upstream nodes to allow forwarded packets to leave the first network structure; as well as The flow remains active within the first network structure after all packets in the flow have left the first network structure and until all packets have reached their destination node.

19. The non-transitory machine-readable storage medium of claim 18, wherein, Instructing the upstream node to indicate that the forwarded packet has left the first network structure includes: Generate and send an acknowledgment packet corresponding to the forwarded packet, the acknowledgment packet specifying the amount of data acknowledged; and The flow range value is updated based on the confirmed packets, and the flow range value is used to track pending packets in the flow within the first network structure.

20. The non-transitory machine-readable storage medium of claim 19, wherein, Maintaining the flow activity includes: suppressing acknowledgments of at least a portion of the initial packets in the flow to ensure that the flow range value is non-zero.

Images