Method and device for congestion control

Abstract

A method and apparatus for congestion control are described. The apparatus may include a network port connected to a network. The apparatus may include a first controller configured to send and receive a first communication on the network using the network port. The apparatus may include a storage device for controller records of the controllers, which may store congestion scores, congestion timestamps, and uncongested timestamps. The apparatus may also include a storage device for a device-wide record, the device-wide record including at least a second congestion score and a second congestion timestamp of the first controller and a third congestion score and a third congestion timestamp of the second controller. The device-wide record may be based at least in part on the controller record. A throttle may limit a second communication of the second controller based at least in part on the device-wide record.

Description

Technical Field

[0001] The present disclosure relates generally to network-attached devices and, more particularly, to methods and apparatus for managing congestion in network-attached storage devices. Background Art

[0002] A network-connected storage device (such as an Ethernet-connected storage device) allows other devices to communicate directly with the storage device, rather than with a processor that may be part of a larger system that includes the storage device. By removing the system processor, memory, and other components from the overall system, power requirements can be reduced. In addition, because components such as processors can be removed from the communication path, the time required before input / output operations are completed can be reduced.

[0003] But as storage devices grow in size, so does the amount of data written to those storage devices. Files and data sets can become larger, and large amounts of data are sent across the networks that connect storage devices to the applications that use the data. Just as the number of cars on a highway increases traffic and can cause congestion (slowing down all traffic), increased network traffic can also cause congestion, which can slow the delivery of information across the network.

[0004] There is still a need for a mechanism to manage network traffic congestion. Summary of the invention

[0005] Embodiments of the present disclosure include devices in a network. These devices may include controllers, and may determine whether the controllers are associated based on sharing some communication paths with another controller. If the controllers are associated, and one controller experiences congestion, the associated controller may also actively apply congestion control. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings described below are examples of how to implement embodiments of the present disclosure and are not intended to limit the embodiments of the present disclosure. Various embodiments of the present disclosure may include elements not shown in a particular drawing and / or may omit elements shown in a particular drawing. The accompanying drawings are intended to provide illustrations and may not be drawn to scale.

[0007] Figure 1 A system including devices connected via a network according to an embodiment of the present disclosure is shown.

[0008] Figure 2 The embodiment according to the present disclosure is shown Figure 1 Details of the equipment.

[0009] Figure 3 The embodiment according to the present disclosure is shown Figure 1Detail of the rack.

[0010] Figure 4 A Non-Volatile Memory Express (NVMe over Fabrics, NVMe-oF) initiator sending a command and an NVMe-oF target responding according to an embodiment of the present disclosure are shown.

[0011] Figure 5 The embodiment according to the present disclosure is shown Figure 1 In the system Figure 4 NVMe-oF Initiator and Figure 4 NVMe-oF target communication.

[0012] Figure 6 The embodiment according to the present disclosure is shown Figure 5 An alternative view of the NVMe-oF target and NVMe-oF initiator communications.

[0013] Figure 7 The embodiment according to the present disclosure is shown Figure 5 A switch that marks packets with congestion notification in communications between an NVMe-oF target and NVMe-oF.

[0014] Figure 8 FIG. 1 shows a method for assembling information for active congestion control according to an embodiment of the present disclosure. Figure 1 equipment.

[0015] Fig. 9 The embodiment according to the present disclosure is shown Figure 2 Details of the controller association matrix.

[0016] Fig.10 The embodiment according to the present disclosure is shown Figure 2 Details of the system level association matrix.

[0017] Fig.11 The embodiment according to the present disclosure is shown Figure 2 The controller records the details.

[0018] Fig.12 The embodiment according to the present disclosure is shown Figure 2 Device-wide logging details.

[0019] Figures 13A-13B The embodiment according to the present disclosure is shown Figure 1 A flowchart that provides a high-level overview of an example of how a device can apply proactive congestion control.

[0020] Fig.14 The embodiment according to the present disclosure is shown Figure 1 An alternative flow chart of an example process for applying active congestion control for a device.

[0021] Figures 15A-15B The embodiment according to the present disclosure is shown Figure 1 Device determination Figure 1 A flow chart of an example process for associating two controllers of a device.

[0022] Fig.16 The embodiment according to the present disclosure is shown Figure 1 Device determination Figure 1 Flowchart of an example process for associating two controllers of a device by having a shared switch.

[0023] Fig.17 The embodiment according to the present disclosure is shown Figure 1 Flowchart of an example process for a device to handle congestion notification.

[0024] Fig.18 The embodiment according to the present disclosure is shown Figure 1 After receiving the congestion notification, the device Figure 2 A flow chart of an example process for a controller to determine a congestion score.

[0025] Fig.19 The embodiment according to the present disclosure is shown Figure 1 Flowchart of an example process for a device to proactively apply congestion control. DETAILED DESCRIPTION

[0026] Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. In the following detailed description, many specific details are set forth to enable a thorough understanding of the present disclosure. However, it should be understood that one of ordinary skill in the art may practice the present disclosure without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail to avoid unnecessarily obscuring various aspects of the embodiments.

[0027] It should be understood that although the terms first, second, etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first module can be referred to as the second module, and similarly, the second module can be referred to as the first module without departing from the scope of this disclosure.

[0028] The terms used in the description of the disclosure herein are only used for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in the description of the disclosure and the appended claims, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any and all possible combinations of one or more related listed items. It will be further understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. The components and features of the accompanying drawings are not necessarily drawn to scale.

[0029] Building efficient networks can be very important for data centers serving big data, artificial intelligence (AI) / machine learning (ML), cloud-based network storage workloads, etc., where maintaining service level agreements (SLAs) for throughput and latency can be important. Non-volatile memory express (NVMe) over fabric (NVMe-oF) technology is a form of storage device used in data centers because NVMe supports parallelism in executing storage commands over a network fabric. Ethernet is a commonly used network protocol deployed in data centers because it is more popular, more economical, and easier to manage. NVMe over Transmission Control Protocol (TCP) (NVMe / TCP) and NVMe over Converged Network Remove Direct Memory Access (RDMA) (NVMe / RoCE) (using any version of RoCE) are currently available commonly used Ethernet-based protocols that support connections between NVMe-oF initiators and their targets. However, although the above (and below) discussion may focus on NVMe storage devices and specific Ethernet-related protocols, embodiments of the present disclosure may be extended to any storage device that supports network connections, regardless of whether a storage protocol or a network protocol is used.

[0030] In some embodiments of the present disclosure, approximately 80% of the traffic generated by modern workloads can remain within the data center: that is, both the initiator and the target can be within the data center. The data center may experience network congestion: the goal of any device in the data center is to recover quickly from network congestion. NVMe targets can be a major cause of congestion because they may be serving multiple initiators in the data center at the same time.

[0031] A mechanism for predicting when the NVMe-oF target can proactively take action to reduce congestion in the data center would help reduce congestion when there are multiple initiators connecting to the same target, or when there are multiple connections from a target to the same initiator.

[0032] The server that exports the NVMe device to the initiator through the fabric is called an NVMe-oF target. The server can be a single server that exports one or more NVMe solid-state drives (SSDs) or Ethernet-connected SSDs. In addition, the target can have one or more network interfaces, each of which can operate independently of other network interfaces. In addition, the target can allow the initiator to open one or more NVMe-oF queue pairs to submit I / O commands. Each NVMe-oF queue pair can provide an independent and parallel network path for the NVMe queues in the storage device. In some embodiments of the present disclosure, commands sent in an NVMe-oF queue pair can be processed by the same queue pair (that is, responses to I / O commands can be sent using the same NVMe-oF queue pair). Each target can support one or more management queue pairs and one or more I / O queue pairs. Management queue pairs may be rarely used; embodiments of the present disclosure can be used only with I / O queue pairs (which can be used for data transmission), only with management queue pairs (which can be used for management of NVMe targets), or with two sets of queue pairs.

[0033] The term NVMe-oF target can refer to the software that manages the storage device and network connection and can help remote initiators access the storage.

[0034] The term NVMe-oF initiator or host may refer to a server that is connected to an NVMe-oF target through a fabric to perform storage operations. The term NVMe-oF initiator may also refer to software that enables applications to access remote NVMe-oF targets. Each NVMe-oF initiator may have a unique host ID. The NVMe-oF initiator may be a single physical server, or a virtual machine (VM) hosted on a physical server. Multiple virtual machines hosted on a single physical server may also be initiators connected to the same NVMe-oF target. Since multiple VMs hosted on a physical server may have different Internet Protocol (IP) addresses and multiple host IDs, these VMs may appear to be different NVMe-oF initiators to the NVMe-oF target. However, since these VMs share the same network interface card (NIC) and internal software switch on the physical server, if the internal switch is congested or the shared NIC may not be able to process incoming packets, some or all of the VMs will be affected.

[0035] The term NVMe-oF controller may refer to an abstract entity that represents a one-to-one relationship between a single initiator network port and a single target network port. Each NVMe-oF controller pair may have an independent set of (multiple) NVMe queue pairs, independent network connections, and may perform parallel I / O operations. By using multiple queue pairs, throughput may be increased through multiple parallel connections or by providing separate paths for I / O commands (e.g., based on their size, priority, etc.).

[0036] The NVMe-oF controller can associate an NVMe queue pair with a network connection to create an NVMe-oF queue pair. An NVMe-oF queue pair based on an Ethernet structure can be associated with a TCP or User Datagram Protocol (UDP) port.

[0037] Network topology

[0038] Embodiments of the present disclosure may be applicable to deployments where NVMe-oF targets and initiators may be arranged in a rack or cluster having a network topology using a 2-layer switching fabric, which may be referred to as a leaf-spine topology (sometimes referred to as a spine-leaf topology or a leaf-and-spine topology). Embodiments of the present disclosure may also be applied to other topologies: however, as the number of switches connecting targets and initiators increases, the number of different paths through the network increases, which has an opposite effect on the ability to predict congestion: as the number of paths through the network increases, the efficacy of predicting congestion decreases.

[0039] In a leaf-spine topology, a rack consists of a group of devices installed in a common cabinet. These devices can be connected to one or more switches located within the rack (commonly referred to as top-of-rack (TOR) switches or leaf switches). These TOR switches are connected to another set of switches that interconnect the racks: in general, these switches can be called the spine and interconnect the racks / leaves. The connection between the leaf switches and the spine switches can operate on a layer 2 (switching) or layer 3 (routing) connection. If every leaf switch is connected to every spine switch, there will always be a path from any device in the topology to every other device. In these embodiments of the present disclosure, the spine switches may not need to be interconnected to each other, and the leaf switches may not need to be interconnected to each other.

[0040] Since there may be multiple devices in a rack, there may be multiple NVMe-oF targets in a rack. Since a single NVMe-oF target may have multiple network ports, multiple network ports in an NVMe-oF target may be connected to a common TOR switch or different TOR switches. Separate NVMe-oF queue pairs from the target to a single or multiple initiators can be switched through different paths determined by the switch.

[0041] Multiple NVMe targets may transmit network packets to the same or different initiators at the same time. Therefore, network congestion may occur on the TOR leaf switch or spine switch.

[0042] By identifying the network addresses of the switches to which the target and initiator are connected, it is possible to determine whether the NVMe-oF target and initiator are connected to the same rack or cluster. For example, there are software tools that can be used to identify the IP addresses of the leaf switches and spine switches used for network connections, where NVMe-oF queue pairs may use these IP addresses. Depending on the location of the target and initiator in the topology, the connection between the NVMe-oF target and the initiator may be at the rack level (i.e., only through the TOR switch) or at the cluster level (through both the TOR switch and the spine switch). Different TOR switches and spine switches can be used to transmit network packets of NVMe-oF queue pairs. In some embodiments of the present disclosure, the switch may use an equal cost multipathing (ECMP) scheme to determine the network path based on a combination of source and destination IP addresses, source and destination TCP / UDP ports, transport protocols, etc. In some embodiments of the present disclosure, the switch may continue to use the same path to route packets between the target and the initiator until network conditions (such as congestion or connection failures) cause a new route to be used.

[0043] Data center network congestion

[0044] Data center networks may require high-bandwidth, low-latency network connections switched through shallow-buffered switches (i.e., switches with small buffers that store network traffic). Network traffic can be unpredictable, prone to bursts, and prone to incast-type network congestion (many destinations sending data to the same host). For example, a single host device can host multiple VMs that can share a network interface to a TOR switch. If a host has problems with incast, the switch connected to the host can become congested, which can affect all connections to the host from different spine switches. TOR switches are more susceptible to congestion than intermediate switches or routers. Similarly, if multiple connections from a destination are routed through the same spine switch, or if servers from the same rack attempt to transmit data to hosts in the same cluster through the same spine switch, the TOR switch serving the destination can become congested.

[0045] If one or more of the connections from a target to a host / cluster is congested, then another connection to the same host / cluster may also be congested.

[0046] Network congestion handling on switches

[0047] A switch may have an internal buffer that can queue certain network packets before forwarding them to the destination. Switches operating at layer 3 may support a feature called explicit congestion notification (ECN) to notify senders and receivers about congestion in the switch. TCP, RoCE, QUIC (not an acronym), etc. may use ECN schemes.

[0048] The ECN scheme is a three point scheme in which a switch can mark packets as experiencing congestion and send the marked packets to the receiver. The receiver can then send a message to the sender indicating the congestion. The sender can then take action to control the congestion. In this scheme, there may be delays in implementing this congestion control due to the lag between the time the switch marks the packet to indicate congestion and the time the sender is notified of the congestion by the receiver. Other congestion control schemes that allow the switch to indicate congestion directly to the sender are also possible.

[0049] Depending on how all associated network devices handle network congestion, a network connection can be lossy (i.e., data may be lost) or lossless (i.e., no data is lost). In the case of severe congestion, a lossy structure may drop packets. The sender can then resend the network packets to the receiver. On the other hand, a lossless structure may have mechanisms for the sender to retain network packets until the congestion is cleared, so that no data is lost. In a lossless structure, network packets may not be dropped, but only delayed.

[0050] In both lossy and lossless structures, in order to obtain the best network performance, the sender may need to be notified or otherwise aware of network congestion and take action as early as possible. If the sender does not take action early in a lossy structure, packets may be lost; if the sender does not take action early in a lossless structure, unnecessary back pressure may be generated, which may cause other performance degradation.

[0051] Modern switches can segregate traffic based on the network priority of the packets. However, if the switch cannot handle the incoming network traffic, all traffic of the same priority going to the same destination switch has the same chance of being congested.

[0052] Network congestion detection and control at the network transport layer may work only on a per-connection basis. This mechanism may not be able to determine the intermediate switches / routers that a network packet may have traversed, as the intermediate switches / routers may change dynamically. Therefore, when one or more of the connections indicates congestion, the transport layer may not group connections going to the same destination and apply congestion control actions equally. Identifying and correcting this problem is a subject of the present disclosure.

[0053] ECN-based schemes (TCP, Data Center TCP (DCTCP), Data Center Quantized Congestion Notification (DCQCN), etc.) can be used by devices that notify of congestion at switches / routers. But there may be other ways to detect congestion on a device that does not require any notification from the switch / router. For example, a device can check network packet loss, packet retransmissions, packet delays, and other network statistics to identify that a connection is experiencing congestion. Although it may not be possible to determine in this case whether the problem is with the switch / router or is simply due to congestion at the endpoint, if multiple connections to multiple initiators exhibit these congestion characteristics, congestion at the switch / router can be inferred.

[0054] NVMe-oF message exchange

[0055] NVMe-oF devices may use an asynchronous message transfer mode. An initiator can send multiple storage I / O commands to a target. These commands can be queued in the target. When the commands are processed and completed by the target, the order of these I / O commands may not be maintained. Once the I / O command is processed, the target may send a response to the initiator. There may be a delay between the time the initiator sends the I / O command and the time the target completes the action and sends a response. At this time, the network conditions in the data center may change.

[0056] Furthermore, there may not be a one-to-one correspondence between I / O commands sent by the initiator and response packets sent by the target. For example, the initiator might send a single I / O command to read a large amount of data: so large that the data may not fit into a single packet in the response. In this case, the target might send multiple packets in response to a single I / O command received from the initiator.

[0057] While the target is processing a command (such as a read request), the queue pair can receive other I / O commands from the initiator and queue them. There can be a continuous flow of data between the initiator and the target.

[0058] As described above, NVMe-oF devices may operate differently from other network devices. An NVMe-oF queue pair may be a logical structure that attaches a network socket (TCP / UDP) to an NVMe queue. An I / O queue pair may be used for data transmission. An initiator may send an I / O command in one NVMe queue pair: in some embodiments of the present disclosure, the target must respond in the same queue pair. If the initiator uses multiple NVMe-oF queue pairs, the target may respond in the same queue pair used for the I / O command.

[0059] In a normal network, all network connections can be treated equally. When an application uses multiple parallel connections for load balancing, the server can use any number of these parallel connections. In case of network congestion, the server can use a reduced number of parallel connections to reduce congestion.

[0060] However, in embodiments of the present disclosure (where responses are sent in the same NVMe queue pair as I / O commands), the target may not be able to reduce the number of queue pairs as network conditions change. The target may need another way to reduce the rate of network traffic sent across the data center network while giving all NVMe-oF queue pairs a chance to send their data. If the initiator does not receive a response in a queue pair, the initiator can resubmit the request, which may result in duplicate work.

[0061] Queue pairs in an NVMe-oF target may not share information about congestion observed at the rack or cluster level. The transport layer that performs congestion control has a scheme that works for the general environment and does not take active congestion control actions. Small closed topologies can benefit from additional actions taken by the application layer based on the specific properties of the application, thereby improving the overall efficiency of the system.

[0062] Target congestion management

[0063] In embodiments of the present disclosure, the focus may be on how the target handles congestion when sending data in response to a read request. However, embodiments of the present disclosure are also applicable to initiators that send write requests to the target. Similarly, while the focus may be on NVMe-oF queue pairs, embodiments of the present disclosure are applicable to any implementation that manages congestion without reducing the number of ways that data can be routed.

[0064] NVMe-oF targets can proactively take action on network connections to reduce congestion in a data center. The determination of whether to take action to reduce congestion can be based on previous congestion reported by other network connections in the same target. NVMe-oF targets may be able to react to congestion before existing ECN schemes can notify the target of congestion.

[0065] The NVMe-oF target can use information available in the NVMe-oF command (such as the host ID and other such data) and the network header (such as the source IP address and other such data) to determine which queue pairs (NVMe-oF controllers) may be associated with the same host, rack, and cluster. Since congestion can occur in queue pairs going to the same host, or in those queue pairs going to the same cluster, this association information can allow congestion to be predicted for specific queue pairs.

[0066] Based on the cumulative congestion score of its queue pair, each NVMe-oF controller can have a congestion score for each Class of Service (CoS) priority. Based on the congestion notification information received by the queue pair, the congestion score can quantify the congestion experienced during the transmission of read commands.

[0067] The priority of the queue pair may be set based on an administrative setting, a socket priority setting, or a Differentiated Services Code Point (DSCP) value set in an IP packet received from an initiator. The obtained priority setting may be mapped to a CoS value, which may be in the range of, for example, 0-5. In some embodiments of the present disclosure, only some possible CoS values ​​may be used for data transmission; in other embodiments of the present disclosure, all possible CoS values ​​may be used for data transmission.

[0068] Congestion scores and information about associated controllers can be used to make probabilistic predictions of congestion, which can allow proactive congestion control actions to be taken at the application layer. The target device can predict and apply rate limits at the NVMe-oF layer over short time intervals based on congestion noticed by associated queue pairs in the same CoS, which can reduce congestion more quickly. Rate limiting can be performed on connections associated with the connection that received the congestion notification: other connections may not be involved. Rate limiting can help improve the overall efficiency of the target by better scheduling I / O transfers from unaffected queue pairs when other queue pairs may be predicted to experience congestion. Send rate limits applied to affected queue pairs can supplement congestion control actions taken by the transport layer.

[0069] A service thread in the target may perform path tracing to identify intermediate switches between the target and the initiator. The IP address of the authenticated initiator may already be available (e.g., from the packet header). Otherwise, this information may be obtained when the initiator establishes the connection. When an NVMe-oF initiator connects to multiple interfaces of a target, these controllers may be grouped into associated controllers and entered into a controller association matrix, which may be labeled M. CIn the controller association matrix M C In the example, a value of 1 may indicate that the two controllers are associated; a value of 0 may indicate that the two controllers are not associated.

[0070] Furthermore, when an NVMe-oF initiator establishes a connection, controllers connected to the same destination (rack or cluster) may be grouped into controllers associated at the system level and entered into a system level association matrix, which may be labeled M Sys In this context, the term system refers to the devices (i.e., targets): Each target can have its own system-level association matrix M Sys . At the system level, the correlation matrix M Sys In , a value of 1 may indicate that the two controllers are associated through a switch; a value of 0 may indicate that the two controllers are not associated through a switch. In addition, the degree of association may be represented, where a value of 1 may represent the maximum possible association (e.g., indicating that the two controllers share all intermediate switches), a value of 0 may represent the minimum possible association (e.g., indicating that the two controllers do not share any intermediate switches), and values ​​between 0 and 1 may indicate some (but not all) shared intermediate switches. For example, the system-level association matrix M Sys The value in can be calculated as the ratio of the number of shared intermediate switches of one (or the other) connection relative to the total number of intermediate switches.

[0071] Each controller may also have a record of its congestion score and timestamp. The controller may store the record with a separate entry for each queue pair supported by the controller. For each queue pair, the record may include the CoS / priority assigned to the queue pair, the reception timestamp of the last packet including the congestion notification (which may include, for example, a packet marked by an ECN scheme), the reception timestamp of the last packet without the congestion notification, and the weighted congestion score. The timestamp of normal network events for each queue pair may be obtained from, for example, asynchronous write data received at the NVMe-oF transport layer. The weights used in the weighted congestion scores may take into account the possibility that a particular queue pair may be affected by congestion in another queue pair, or the possibility that congestion in a queue pair may affect another queue pair as a factor. The weight attached to the congestion may be expected to be reduced in proportion to the frequency of congestion in the queue pair. By making the weight proportional to the frequency of congestion in the queue pair, the mechanism can isolate queue pairs / controllers that may frequently experience congestion but may not extend to other connections.

[0072] Whenever a congestion notification is received, the cumulative congestion score for each priority of each controller can be calculated. The cumulative congestion score can be calculated only for queue pairs that have experienced congestion: that is, queue pairs for which the timestamp of the last packet with a congestion notification is more recent than the timestamp of the last packet without a congestion notification. Queue pairs that are not currently experiencing congestion may not be included in the cumulative congestion score.

[0073] In addition, the cumulative congestion score can be calculated for queue pairs for which the timestamp of the last packet with a congestion notification is within a certain predetermined amount of time. For example, if the queue pair for which the most recent packet includes a congestion notification was received an hour ago, the fact that the queue pair experienced congestion an hour ago may not necessarily reflect that the queue pair is currently experiencing congestion. Example intervals for discarding old congestion notifications from the calculation of the cumulative congestion score can include any desired length of time, such as three seconds. The congestion score can also be cleared periodically at any desired time interval, such as after 5 minutes, 15 minutes, or 30 minutes.

[0074] Once the cumulative congestion score for a queue pair has been calculated, the cumulative congestion score can be used to update the record of the target range. The record of the target range can represent a matrix for each controller and each CoS / priority. In the record of the target range, for each controller, for each priority, and for each congestion score, the timestamp of the last packet with a congestion notification and a vector of the weighted congestion scores of the controller can be stored. The vector of weighted congestion scores can be taken from the record of the congestion scores of each controller.

[0075] Given controller i and CoS / priority p, the cumulative congestion score of the associated controller at the initiator level can be calculated as M c [C i ×V p : that is, the row for controller i in M C multiplied by the vector containing the congestion scores corresponding to priority p (taken from the record of the target range). The vector containing the congestion scores can be defined as a function of the weighted congestion scores and a measure of how long ago the last congestion notification was received: V p = wc p ⊙ f(tan p ), where, where, 0 < i < m, where m is the number of controllers. ⊙ can be the Hadamard product, where each element of one vector is multiplied by the corresponding element in the second vector to create another vector of the same dimension. The function f(tcn p) can produce a vector consisting of m components, where each element will be 0 or 1 depending on whether the controller's latest congestion notification timestamp occurred within the past three seconds. Therefore, the function f(tcn p ) can be used to effectively limit the calculation of the weighted congestion score to periodic intervals. (Alternatively, V p can be represented as a vector, where the i-th component v i,p for Indicates that the Hadamard product or function f(tcn p ) is the same concept as ).

[0076] The selection of three seconds in the above calculation is based on the TCP retransmission timeout, which is initially set to three seconds. The principle behind the TCP retransmission timeout is that if there is packet loss, TCP will be aware of this fact after three seconds. This TCP retransmission timer can be used when the transport protocol has not received any ECN and determines that there have been lost packets. During this three-second interval, if the associated controller has received a congestion notification, the associated controller can take proactive measures to avoid congestion. However, the use of three seconds as such an interval is merely an example, and other intervals may be used. In addition, although the above examples focus on TCP as a protocol, embodiments of the present disclosure may be used with other protocols, and other intervals may also be used.

[0077] While the discussion above focuses on lossy structures, similar timing can be used with lossless structures to reduce backpressure.

[0078] Similarly, given controller i and CoS / priority p, the cumulative congestion score of the associated controller at a rack / cluster can be calculated as M sys [C i ]×V p :That is, M Sys The row in V for controller i is multiplied by the vector containing the congestion scores corresponding to priority p (taken from the target range of records): p It can be calculated as described above.

[0079] As described above, the mechanism can limit the applicability of congestion in the relevant queue pairs / controllers based on how long the congestion is expected to remain. Congestion control can be applied for a period of time based on the estimated transmission time, starting from the time the decision to apply congestion control is made. That is, congestion control can be applied for the amount of time it takes the network protocol stack to recognize congestion and take action to correct the congestion. For example, this amount of time can be estimated as the roundtrip time of the network (the time it takes for a packet to travel from its source to its destination and receive a response back at the source). The period can depend on a variety of factors, such as the level at which congestion is observed (same host / multipath or rack / cluster level), the congestion score, and the average queuing delay across the network stack. The average queuing delay can be calculated as Where μ may represent the service rate (the rate at which packets are sent), and λ may represent the packet arrival rate at the network layer of the target's network stack. In other words, λ may represent the total number of packets per second from the various controllers that need to be sent from the target to the various hosts. μ may be obtained from system analysis, and λ may be estimated using input from network statistics tools. During the application of congestion control, additional congestion notifications may not be checked (although they may be checked and used to reset the time period specified for congestion control). One method of calculating this congestion control period may be (TQ delay )×f(C)×g(L), where T can represent the round-trip time, Q delay may denote the average queuing delay, f(C) may denote a function of the congestion score (and may be used to determine the portion of time that congestion control should be applied based on the severity of the congestion), and g(L) may denote a function of the affinity level (such as the controller affinity matrix M referred to above). C and / or the system-level correlation matrix M Sys described above).

[0080] During congestion control, the NVMe payload in each protocol data unit (PDU) from the affected queue pair can be limited to the maximum segment size (MSS) (a value that can be set at the protocol layer). Multiple PDUs may be required to send the entire payload from the affected queue pair, in which case the inter-PDU delay can be calculated as where d total may represent the length of the time period during which congestion control may be applied to the queue pair, and n PDU can represent the number of PDUs needed to send the entire payload: i.e., for TCP (as an example), More generally, it can be expected that the PDU is no larger than the maximum transmission unit (MTU) of the network interface card.

[0081] In the above discussion, the focus is on congestion control at the target. But congestion control can also be applied to the initiator. Congestion control at the initiator may not be as critical as congestion control at the target because write commands (which may involve a large amount of data moving from the initiator to the target) may be less frequent than reads (which may involve a large amount of data moving from the target to the initiator), and the number of targets that a host can use to write data simultaneously may be low compared to the number of hosts attempting to read from the target simultaneously. A target can be identified based on its NVMe Qualified Name (NQN) and the corresponding IP address.

[0082] Congestion control for reads at the target and congestion control for writes at the initiator can be managed independently. Network switches can have separate buffers for sending and receiving data, so congestion during reads at the target end may not mean congestion during writes from the initiator, and vice versa. In addition, when there are multiple network paths, the series of switches that a packet passes through from the target to the initiator can be different from the series of switches that a packet passes through from the initiator to the target.

[0083] Figure 1 A system including devices connected via a network according to an embodiment of the present disclosure is shown. Figure 1 , devices 105-1 to 105-15 are shown organized into three racks 110-1 to 110-3 and connected to form a network 115. Embodiments of the present disclosure may include any number of devices organized into any number of racks. Figure 1 Five devices are shown in each of racks 110 - 1 through 110 - 3 , but embodiments of the present disclosure may have any number (one or more) of racks, and different racks may include different numbers of devices.

[0084] Devices 105-1 to 105-15 may be any type of device that may be used with racks 110-1 to 110-3. Examples of devices 105-1 to 105-15 may include, but are not limited to, storage devices, network interface cards, processors, accelerators, etc. In the following discussion, some of devices 105-1 to 105-15 may be considered storage devices accessible via network 115.

[0085] Racks 110-1 through 110-3 may include switches 120-1 through 120-6. Switches 120-1 through 120-6 may be referred to as top-of-rack switches because they are located at the "top" of the rack and connect to devices in the rack. For example, switches 120-1 and 120-2 are connected to devices 105-1 through 105-5, switches 120-3 and 120-4 are connected to devices 105-6 through 105-10, and switches 120-5 and 120-6 are connected to devices 105-11 through 105-15. Note that switches 120-1 through 120-6 may provide multiple connections to devices 105-1 through 105-15: for example, device 105-1 is connected to both switches 120-1 and 120-2 (although device 105-1 may be connected to switches 120-1 and 120-2 through different ports, as described below with reference to Figure 2 discussed). Although Figure 1 Racks 110-1 through 110-3 are shown, each having two switches, and switches 120-1 through 120-6 are each connected to all devices in the corresponding rack, but embodiments of the present disclosure may include any number (one or more) of switches in a rack (and different numbers of switches in different racks), and a switch in a rack may not be connected to all devices in the rack.

[0086] In addition to switches 120-1 to 120-6 in racks 110-1 to 110-3, network 115 may also include switches 120-7 and 120-8. Switches 120-7 and 120-8 may interconnect switches 120-1 to 120-6 even in different racks, thereby providing one or more paths interconnecting devices 105-1 to 105-15.

[0087] Although the term switch is used herein, the term switch should be understood to include other intermediate elements that can perform similar functions. Therefore, the term switch should be understood to include other elements, such as routers, gateways, etc.

[0088] Figure 1The topology shown in can be referred to as a leaf-spine topology (sometimes referred to as a spine-leaf topology or leaf-spine topology): switches 120-7 and 120-8 can be considered the spine of network 115, and switches 120-1 through 120-6 can be considered the leaves of network 115 (devices 105-1 through 105-15 can be connected to these leaves). It can be seen that any two devices in network 115 can be connected through one or three of switches 120-1 through 120-8: if both devices are in the same rack and connected to the same top-of-rack switch, they are connected through one switch, otherwise they are connected through three switches. (If two top-of-rack switches in the same rack (such as switches 120-1 and 120-2) are connected to each other, the devices in that rack can also be connected using two switches). Although Figure 1 The network 115 is shown arranged in a leaf-spine topology, but embodiments of the present disclosure may be extended to any desired topology.

[0089] Although Figure 1 It is implied that network 115 is a local area network (LAN), such as may be used to interconnect devices 105-1 to 105-15, such as may occur within a data center, but network 115 may be extended to other forms of networks, such as wide area networks (WANs), metropolitan area networks (MANs), and global networks such as the Internet. However, as described below, the benefits of embodiments of the present disclosure may be greater in networks with fewer paths between devices in the network.

[0090] In the remainder of this document, any reference to elements of the drawings will use common reference numerals. For example, any reference to "device 105" may be understood as a reference to any of devices 105-1 to 105-15, any reference to "rack 110" may be understood as a reference to any of racks 110-1 to 110-3, and any reference to "switch 120" may be understood as a reference to any of switches 120-1 to 120-8.

[0091] The rack 110 may also include Figure 1Various other components not shown in the figure. For example, the rack 110 may include one or more processors and / or memories that can be used to manage the functions of the rack 110 (rather than necessarily being accessible via the network 115). Such processors can be any kind of processor. Each of these processors can be a single-core or multi-core processor, each of which can implement a reduced instruction set computer (RISC) architecture or a complex instruction set computer (CISC) architecture (etc.), and can be mixed in any desired combination. The memory can be any kind of memory, such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), permanent random access memory, ferroelectric random access memory (FRAM) or non-volatile random access memory (NVRAM) (such as magnetoresistive random access memory (MRAM), etc.). The memory may also be any desired combination of different memory types and may be managed by a memory controller. The processor and memory may also support an operating system under which various applications may run. These applications may issue requests (also referred to as commands) to read data from or write data to the memory or device 105.

[0092] Figure 2 The embodiment according to the present disclosure is shown Figure 1 Details of the device 105. Figure 2 , device 105 is shown. Device 105 may include ports 205-1 and 205-2, through which device 105 may communicate with switch 120. For example, if device 105 is Figure 1 In rack 110-1, port 205-1 can connect device 105 to Figure 1 switch 120-1, and port 205-2 can connect device 105 to Figure 1 Switch 120-2. Figure 2The device 105 is shown to include two ports 205-1 and 205-2, but embodiments of the present disclosure may include any number (one or more) of ports 205 to connect the device 105 to the switch 120. In addition, the number and Figure 1 In rack 110 Figure 1 The number of rack top switches 120 does not need to correspond: the device 105 may include more than Figure 1 In rack 110 Figure 1 The number of ports 205 of the rack top switch 120 is less or greater. (Of course, if the device 105 includes more than Figure 1 If the number of ports of the top-of-rack switch 120 is greater, some ports may not be used, and if the device 105 is Figure 1 The rack 110 includes Figure 1 The number of ports of the top-of-rack switch 120 is less, and the device 105 may not be connected to Figure 1 In rack 110 Figure 1 All top-of-rack switches 120).

[0093] Associated with each port 205 may be one or more controllers. Figure 2 , device 105 is shown as a Non-Volatile Memory Express (NVMe) over Fabric (NVMe-oF) device: Thus, NVMe-oF controllers 210-1 and 210-2 may be associated with port 205-1, and NVMe-oF controllers 210-3 and 210-4 may be associated with port 205-2. Figure 2 Two NVMe-oF controllers 210 are shown associated with each port 205, but embodiments of the present disclosure may include any number (one or more) of NVMe-oF controllers 210 associated with each port 205, and the number of NVMe-oF controllers 210 associated with each port 205 may be different. In addition, if the device 105 supports a method other than NVMe-oF for transmitting data over the network, the device 105 may be configured to communicate with the NVMe-oF controller 210 through the NVMe-oF interface. Figure 1 If the protocol used for communication on the network 115 is different from that of the NVMe-oF controller, the controller 210 may be a controller of a different type than the NVMe-oF controller.

[0094] Device 105 may also include storage device 215, which may be used to store various information. The information stored in storage device 215 may include controller association matrix 220, system level association matrix 225, controller record 230, and device range record 235. Figure 9-12 Controller association matrix 220, system level association matrix 225, controller record 230, and device scope record 235 are discussed further. Figure 2Storage 215 is shown as a single storage that includes all of controller association matrix 220, system-level association matrix 225, controller records 230, and device-scope records 235, but embodiments of the present disclosure may include two or more different storages that may store any desired subset of controller association matrix 220, system-level association matrix 225, controller records 230, and device-scope records 235. Furthermore, in embodiments of the present disclosure where device 105 is a storage device, device 105 may include any number (one or more) of storage units (such as hard disk drives or solid state drives (SSDs)) that together form a single target: controller association matrix 220, system-level association matrix 225, controller records 230, and device-scope records 235 may apply to device 105 as a unit, rather than to individual storage units within device 105. However, embodiments of the present disclosure may include device 105 that includes copies of controller association matrix 220, system-level association matrix 225, controller records 230, and device-scope records 235 for multiple targets and / or different storage units.

[0095] Device 105 may also include a path tracker 240 and a throttle 245. Path tracker 240 may be used to track Figure 1 The path used by another device 105 in the network 115 to communicate. A tool that can be used as a path tracer 240 can, for example, identify the path that connects the device 105 to the Figure 1 Another device in network 115 is connected Figure 1 The network addresses of all switches 120. These tools can be accessed from Figure 1 Device 105 or other component of network 115 (e.g., Figure 1 120) to capture target information. In addition, path tracker 240 may be able to identify information such as the network addresses of other devices with which device 105 may communicate. Figure 9-10 As discussed, this information may be used to determine whether controllers 210 - 1 and 210 - 2 may be associated.

[0096] The throttle 245 may be used to control the data sent through the controller 210, where the controller is being actively controlled to prevent congestion. That is, assume that the controller 210-1 is identified as being congested, as described below with reference to Figure 7As discussed, controllers 210-2 to 210-4 can be actively controlled to prevent congestion from occurring on these controllers. The throttle 245 can operate to limit the size of packets sent via the controllers 210-2 to 210-4, or to limit the frequency at which packets can be sent. For example, if the throttle 245 limits the size of packets, the throttle 245 can limit the packets to the maximum segment size (a parameter associated with the Transmission Control Protocol (TCP)) or some other appropriate maximum transmission unit (MTU) that can be used in network communications. Alternatively, if the throttle 245 limits the frequency of sending packets, the throttle 245 can determine the inter-packet delay, which can be calculated as the ratio of the amount of data to be sent within a given interval to the maximum size of each packet. Note that the throttle 245 can apply multiple restrictions: limiting the packet size and limiting the packet frequency can be combined. By using such a technique, the throttle 245 can prevent the controllers 210-2 to 210-4 from experiencing congestion. The throttle 245 can apply active congestion control for any desired period of time. For example, throttle 245 may apply active congestion control for the amount of time required for a round-trip communication from device 105 to another device and back again: other periods may also be used. The period over which throttle 245 may apply active congestion control may factor in where the congestion occurs: at port 205, at some switch between device 105 and another device, a fraction reflecting the level of congestion experienced by controller 210-1 (this will be referred to below). Fig.11 further discussion) and the average queuing delay (which can be expressed as where μ may represent the service rate (the average rate at which packets are sent), and λ may represent the average packet arrival rate).

[0097] Device 105 may also include Figure 2 Other components not shown may depend on the functionality provided by device 105. For example, if device 105 is a storage device, device 105 may also include storage means (such as a hard drive or flash memory) and a controller to manage reading and writing data from the storage device. If device 105 is a local processor or accelerator, device 105 may include circuits and / or software to implement the supported functionality.

[0098] Figure 3 The embodiment according to the present disclosure is shown Figure 1 Details of the rack 110. Figure 3Typically, the rack 110 includes one or more processors 305, which may include a memory controller 310 and a clock 315, which may be used to coordinate the operation of machine components. The processor 305 may also be coupled to a memory 320, which may include, for example, a random access memory (RAM), a read-only memory (ROM), or other state retention media. The processor 305 may also be coupled to a storage device 325 and a network connector 330, which may be, for example, an Ethernet connector or a wireless connector. The processor 305 may also be connected to a bus 335, which may be attached to a user interface 340 and input / output (I / O) interface ports, etc., which may be managed using an I / O engine 345.

[0099] Figure 4 The embodiment according to the present disclosure is shown Figure 1 The NVMe-oF initiator 105-1 sends a command and Figure 1 The NVMe-oF target 105-2 responds. Figure 4 In the example, the NVMe-oF initiator 105-1 may send a command protocol data unit (PDU) 405. The NVMe-oF initiator 105-1 may also be referred to as a host. In response to this command PDU 405, the NVMe-oF target may send multiple PDUs, including data PDUs 410-1 to 410-3 and a response PDU 415. (Although Figure 4 The NVMe-oF target 105-2 is shown sending three data PDUs 410-1 to 410-3, but embodiments of the present disclosure may support any number of data PDUs sent by the NVMe-oF target 105-2.) Thus, a single packet sent by one device 105 may result in another device 105 sending multiple packets in response.

[0100] As described above, device 105 may be an NVMe-oF device. Although embodiments of the present disclosure may include devices using other protocols, the NVMe-oF protocol includes some features that may enhance the benefits of embodiments of the present disclosure.

[0101] exist Figure 5 In the embodiment of the present disclosure, the NVMe-oF initiator and Figure 1NVMe-oF target communication in the system. As the name implies, the NVMe-oF initiator can be an NVMe-oF device that initiates a request to the NVMe-oF device (NVMe-oF target). Using NVMe-oF, the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 can each have multiple queue pairs (each queue pair can include a submission queue for receiving packets and a completion queue for sending packets). There can be one (or more) queue pairs for management purposes and one (or more) queue pairs for input / output (I / O) purposes.

[0102] When establishing a connection between the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2, the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 each use a specific queue pair, as represented by a "channel" connecting the queue pair (which may include communication via, for example, TCP). When sending communications via a specific queue pair, the NVMe-oF protocol expects that responses will be sent using the same queue pair. This expectation has the effect of preventing the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 from being able to communicate using alternative paths: communications between them are expected to follow the same path.

[0103] Figure 6 helps illustrate this point. Figure 6 , the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 are communicating. The NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 may include NVMe-oF layers 605-1 and 605-2, and TCP layers 610-1 and 610-2, respectively. The NVMe-oF layers 605-1 and 605-2 may be used to package / unpack data using the NVMe-oF protocol, and the TCP layers 610-1 and 610-2 may be used to package / unpack data using the TCP protocol. Whichever switch, such as the switch 120-1, may be along the path for sending packet communications between the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2, the switch may be expected to be part of the path used for any other packet communications between the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2.

[0104] If the switch 120-1 between the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 (or, in a similar manner, the switch 120-1 between the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 Figure 2When also considering a leaf-spine topology (which may provide a limited number of paths between the NVMe-oF initiator 105-1 and the NVMe-oF target 105-2 (especially if the communication may not be switched to other ports of the device 105)), the switch 120-1 (or Figure 2 Congestion in port 205 of the same switch 120-1 (or Figure 2 These considerations can be compared with other network protocols or other network topologies, where the other network protocols can allow traffic to propagate via alternative paths, where the other network topologies can provide more Figure 1 115, and may make embodiments of the present disclosure potentially less effective (because Figure 1 Congestion at a particular point within network 115 is unlikely to affect other communications that may traverse that point).

[0105] But to understand the benefits of active congestion control, it helps to understand how congestion control can be applied in other ways. Figure 7 Switch 120 - 1 is shown implementing a scheme such as Explicit Congestion Notification (ECN): Embodiments of the present disclosure may support other congestion notification schemes besides ECN.

[0106] The NVMe-oF target 105-2 may send data intended for the NVMe-oF initiator 105-1, as shown in packet 705. Figure 7 In FIG. 1 , NVMe-oF target 105-2 is shown as sending data because a read operation may be faster than Figure 1 The write operations in the network 115 are more frequent, so when the NVMe-oF target 105-2 attempts to return data to the NVMe-oF initiator 105-1 in response to the read command, congestion may be more likely to occur. However, the embodiments of the present disclosure are also applicable to the NVMe-oF initiator 105-1 sending write data to the NVMe-oF target 105-2).

[0107] In ECN, when the switch 120-1 determines that there is congestion at the switch 120-1, the switch 120-1 may set a bit in the packet, shown as a congestion notification 710 in the packet sent from the switch 120-1 to the NVMe initiator 105-1. Upon receiving the packet 705 with the congestion notification 710, the NVMe-oF initiator 105-1 may send an acknowledgment packet 715 back to the NVMe-oF target 105-2: the acknowledgment packet 715 may include an echo of the congestion notification, shown as a congestion notification echo 720.

[0108] However, it can be seen from this explanation that the NVMe-oF target 105-2 is not aware of the congestion until the NVMe-oF target 105-2 sending data receives the congestion notification echo 720 from the NVMe-oF initiator 105-1. During this period, the NVMe-oF target may have sent more data, which may aggravate the congestion at the switch 120-1 and may cause congestion of other communications passing through the switch 120-1.

[0109] Once the NVMe-oF target 105-2 receives the congestion notification echo 720, the NVMe-oF target 105-2 may take action to help reduce congestion in communications between the NVMe-oF target 105-2 and the NVMe-oF initiator 105-2: such actions may include reducing the amount of data sent by the NVMe-oF target 105-2 or the frequency with which data is sent from the NVMe-oF target 105-2. However, to further help address this situation, and to reduce the likelihood that other traffic passing through the switch 120-1 may become congested, embodiments of the present disclosure may involve the NVMe target 105-2 considering whether active congestion control should be applied to other communications passing through the switch 120-2. Devices such as the NVMe-oF target 105-2 may not be able to identify congestion as Figure 1 The NVMe-oF target 105-2 may be a result of traffic between other devices in the network 115, but the NVMe-oF target 105-2 may be able to take proactive congestion control based on congestion identified in another connection involving the NVMe-oF target 105-2. Figure 8-12 It is described how this identification is performed.

[0110] Figure 8 FIG. 1 shows a method for assembling information for active congestion control according to an embodiment of the present disclosure. Figure 1 Device 105. Figure 8 In the first phase, referred to as the connection phase, the device 105 can identify the same host communicating with the same host. Figure 2 Controller 210. Figure 2 Such controllers 210 may be considered controller-associated, and the controller association matrix 220 may be used to identify Figure 2 Which controllers 210 are so associated. In addition, in some embodiments of the present disclosure, by Figure 2 Two controllers communicating on the same port 205 , whether communicating with the same host or with different hosts, may also be considered associated and marked as such in the controller association matrix 220 .

[0111] Device 105 can also identify the different hosts communicating with Figure 2Controller 210, but different hosts are in the same rack or cluster. Figure 2 Such controllers 210 are said to be system-associated, and the system-level association matrix 225 can be used to identify Figure 2 Which controllers 210 are so associated.

[0112] Note that after identifying the use Figure 2 205, with the same host, or with hosts in the same rack or cluster Figure 2 After all controllers 210 are connected, what remains is the communication with different hosts in different racks or clusters. Figure 2 Controller 210. Figure 2 This controller 210 does not have Figure 2 associated with any other controller 210.

[0113] From the packet used to establish communication with the host, the device 105 can extract a differentiated services field codepoint (DSCP), which can provide some information about the priority or service level (CoS) of the communication. For example, in some embodiments of the present disclosure, the DSCP value can be in the range of 0-63 and can be mapped to a CoS value by a switch. In some embodiments of the present disclosure, the CoS can be in the range of 0 to 5, with 0 representing the lowest CoS and 5 representing the highest CoS; other embodiments of the present disclosure can use other values, which may or may not include consecutive integer values. In other embodiments of the present disclosure, a virtual local area network (VLAN) packet can be used to determine the CoS used to communicate with the host: the CoS value can be determined based on the priority code point (PCP) of the VLAN packet.

[0114] The CoS information, in conjunction with the queue pair associated with the communication, can be used to update the controller record 230. Note that while the controller affinity matrix 220 and the system-level affinity matrix 225 are set during the connection phase and then may not change (until a new connection is established or an old connection is ended), the controller record 230 can be updated during communication with the host. Therefore, the controller record 230 can be considered part of the transmission phase.

[0115] As described above, the controller record 230 can be updated during the communication. Specifically, the controller record 230 can be updated with information about the packets delivered to the device 105 as part of the communication. For example, the information can include update timestamps when packets marked with congestion notifications arrive and when normal packets (i.e., packets not marked with congestion notifications) arrive. In addition, when packets marked with congestion notifications arrive, the congestion score for the queue pair can be determined. The controller record 230, the controller association matrix 220, and the system-level association matrix 225 can then be used to determine the information in the device-wide record 235, and then it can be determined whether the associated controller should be subject to active congestion control.

[0116] Fig. 9 The embodiment according to the present disclosure is shown Figure 2 Details of the controller association matrix 220. Fig. 9 In FIG. 1 , the controller association matrix 220 is shown as a matrix, Figure 1 The value 1 may indicate that the Figure 2 The two controllers 210 are associated, and a value of 0 may indicate Figure 2 The two controllers 210 are not associated. Embodiments of the present disclosure may use any value to indicate association or not: the values ​​1 and 0 are just examples.

[0117] Communicating with the same host Figure 2 Any two controllers 210 can be marked as associated in the controller association matrix 220. And in some embodiments of the present disclosure, using Figure 2 The same port 205 Figure 2 The two controllers 210 may also be marked as associated in the controller association matrix 220 .

[0118] Note that the associations in the controller association matrix 220 are symmetric, commutative, and transitive. Symmetry means that Figure 2 Any controller 210 can be associated with itself: all entries along the main diagonal of the controller association matrix 220 can be 1. The symmetry can be important because Figure 2A particular controller 210 may include more than one queue pair, and congestion on one queue pair may or may not affect the likelihood of congestion on another queue pair in that controller. Commutativity means that if controller 1 is associated with controller 2, then controller 2 must be associated with controller 1. Commutativity can be seen from the symmetry of the controller association matrix 220 about the main diagonal. Finally, transitivity means that if controller 1 is associated with controller 2, and controller 2 is associated with controller 3, then controller 1 is also associated with controller 3. Transitivity shows that for any two controllers that are associated, the rows and columns representing those controllers can be the same.

[0119] Fig.10 The embodiment according to the present disclosure is shown Figure 2 Details of the system-level correlation matrix 225. Fig.10 In FIG. 2 , the system level correlation matrix 225 is shown as a matrix, Figure 1 The value 1 may indicate that the Figure 2 The two controllers 210 are associated, and a value of 0 may indicate Figure 2 The two controllers 210 are not associated. Embodiments of the present disclosure may use any value to indicate association or not: the values ​​1 and 0 are just examples.

[0120] Communicates with hosts in the same rack or cluster Figure 2 Any two controllers 210 may be marked as associated in the system level association matrix 225. Note that Fig.10 It can be concluded that Figure 2 The two controllers 210 are communicating with different hosts, because otherwise the hosts would not be in different racks / clusters.

[0121] Similar to Fig. 9 Controller association matrix 220, the associations in system-level association matrix 225 may be symmetric and commutative: However, the associations in system-level association matrix 225 are not necessarily transitive. For example, temporarily returning to Figure 1Consider a scenario where device 105-1 can communicate with device 105-6 via switches 120-1, 120-7, and 120-3, device 105-1 can communicate with device 105-11 via switches 120-1, 120-8, and 120-5, and device 105-7 can communicate with device 105-12 via switches 120-4, 120-8, and 120-12. Switch 120-1 is used for the first and second communication paths, and switches 120-8 and 120-5 are both used for the second and third communication paths. However, there is no common port or switch between the first and third communication paths. Therefore, while the first and second communication paths can be associated, and the second and third communication paths can be associated, the first and third communication paths are not associated, and therefore Fig.10 The system-level correlation matrix 225 is not transferable.

[0122] Although Fig.10 The system-level association matrix 225 is shown using only values ​​1 and 0, but embodiments of the present disclosure may also support fractional values. Fractional values ​​may be used to represent the degree of association. For example, counting all of the ports and switches used in the communication paths, the number of common ports and / or switches may be divided by the number of ports and / or switches in each communication path (or, if the number of components in the paths is different, the number of ports and / or switches in the paths that communicate with more such components). Thus, continuing with the example above that proves that the system-level association matrix 225 is not transitive, because the first and second paths each include five components (two ports and three switches each), and the first and second paths have one component (switch 120-1) in common, the system-level association matrix 225 may reflect the degree of association as 1÷5=0.2; similarly, the degree of association of the second and third paths may be calculated as 2÷5=0.4. In these embodiments of the present disclosure, the degrees of association reflected in the system-level association matrix 225 can span any desired range and are not necessarily limited to values ​​between 0 and 1; however, it is useful for one end of the range to correspond to values ​​indicating no association and the other end of the range to correspond to values ​​indicating maximum association.

[0123] Fig.11 The embodiment according to the present disclosure is shown Figure 2 The controller records the details of 230. Fig.11 , the controller record 230 is shown as a matrix. However, Fig. 9 The controller association matrix 220 or Fig.10Unlike the system-level association matrix 225 of FIG. 2 , the controller record 230 stores information about a given queue pair managed by a particular controller and may not associate information about those queue pairs. Instead, the controller record 230 may store information such as the CoS associated with each queue pair, the weighted congestion score for each queue pair, the timestamp when the last packet tagged with a congestion notification was received, and the timestamp when the last packet not tagged with a congestion notification was received. Fig.11 , the controller record 230 is shown as storing information about n queue pairs: In an embodiment of the present disclosure, the number of queue pairs managed for a particular controller may vary depending on the controller. Note that since the controller record 230 stores information about queue pairs associated with a particular controller, and in Figure 1 There may be any number of devices 105 Figure 2 The controller 210, therefore Figure 1 In the device 105 Figure 2 There may be one controller record 230 for each controller 210. Alternatively, if the controller record 230 can identify a queue pair and a controller associated with the queue pair, only one controller record 230 may be stored.

[0124] Among the data stored in the controller record 230, the weighted congestion score deserves special discussion. The weighted congestion score is intended to indicate how congested this particular queue pair is, and how likely it is that congestion on this queue pair will affect other controllers. The weighted congestion score may initially be set to a constant (when the first packet marked with a congestion notification arrives). The fact that subsequently arriving packets are also marked with congestion notifications may indicate that the queue pair is experiencing congestion that may not necessarily affect other queue pairs in the controller. Therefore, when subsequent packets also marked with congestion notifications arrive, the congestion score may be weighted by taking the inverse of the number of packets marked with congestion notifications received for the queue pair. Mathematically, the weighted congestion score may be calculated as Where K may represent a constant used to initially set the congestion score, and p may represent the number of packets received for the queue pair that are marked with congestion notifications.

[0125] With respect to other data stored in the controller record 230, as described above, the congested and uncongested timestamps are the times when the most recent packet was received with and without congestion notification, respectively. Figure 1timestamp within device 105. The CoS may depend on the connection to the host and may change over time: a single controller may use different CoSs for different connections over time, and may be set by the host or switch (although in some embodiments of the present disclosure, the target may set the CoS). (However, in some embodiments of the present disclosure, the CoS assigned to a queue pair may be fixed for the duration of the connection to the host). Note that different queue pairs may be assigned the same CoS: if Figure 2 The fact that the controller 210 supports more queue pairs than service categories is expected. In other words, there is no correlation between the number of service categories and the number of queue pairs.

[0126] In some embodiments of the present disclosure, the information in the controller record 230 may be permanently stored, with new information being added over time. However, over time, information about past congestion events becomes less and less relevant, and at some point may be irrelevant to current congestion management. Therefore, in some embodiments of the present disclosure, Figure 1 The device 105 can track how long the information in the controller record 230 has been present. In particular, Figure 1 The device 105 may track how much time has passed since the controller record 230 was first updated to reflect the receipt of a packet tagged with a congestion notification. After this time interval has passed (which may be measured by setting a timer, or by tracking the timestamp of when the first packet tagged with a congestion notification was received and comparing that time to the current time, and may span any desired amount of time, such as 10 minutes, 30 minutes, 1 hour, etc.), the controller record 230 may be erased. Figure 2 The controller 210 of FIG. 210 includes information about a particular queue pair or all queue pairs, a weighted congestion score and timestamp for discarding one or more queue pairs, and the number of packets marked with congestion notifications for the queue pairs (note that the CoS may be retained because the CoS may not change until the connection with the host ends). In some embodiments of the present disclosure, this tracking of the first packet marked with congestion notification may occur at the device level, rather than the controller level: once the interval has passed, Figure 2 The controller records 230 of all controllers 210 may be erased. Note that this interval used to track congestion may be different from Figure 2 The throttle 245 may apply periods of active congestion control.

[0127] However, if the controller record 230 is erased in this manner, and the queue pair has just experienced congestion, erasing the controller record 230 may result in Figure 2 The controller 210 or Figure 2Other controllers 210 of the present invention miss the opportunity for proactive congestion control when such action might be beneficial. In such a case, where the most recently marked packet with a congestion notification is within a certain delta at the end of the interval, the number of packets marked with a congestion notification (i.e., the weight) can be reset to 1 instead of 0 (the unweighted congestion score can remain constant K). In this way, even after the end of the interval, the associated controller can still apply proactive congestion control if appropriate. The delta can be measured as a percentage of the interval (e.g., 5%) or can be a measured amount of time (e.g., 2 minutes).

[0128] Fig.12 The embodiment according to the present disclosure is shown Figure 2 Details of the device range record 235. Fig.12 , the device range record 235 is shown as a matrix. Similar to the controller record 230, and different from Fig. 9 The controller association matrix 220 or Fig.10 In the system-level association matrix 225, the device-scoped record 235 stores information about the controllers, but does not associate information about the controllers. Instead, the device-scoped record 235 may store information about congestion associated with each controller as a whole.

[0129] In device scope record 235, for each controller and for each CoS, the Fig.11 The timestamp and weighted congestion score of the most recently received packet marked with a congestion notification are extracted from the controller record 230 of Figure 2 A given controller 210 may identify Fig.11 The corresponding controller record 230. Then, for a given CoS, you can check Fig.11 Each queue pair in the controller record 230 is considered to identify its CoS: only queue pairs with the CoS of interest are considered (for other CoSs, other queue pairs may be considered). Across the queue pairs with the CoS of interest, the timestamp of the most recently received packet marked with a congestion notification may be copied to the device-wide record 235 along with the weighted congestion score from the same queue pair. Note that the weighted congestion scores from other queue pairs with older congestion timestamps may be ignored, as in the expression V above. p =wc p ⊙f(tcn p ), where In some embodiments of the present disclosure, the Fig.11The controller records 230 the congestion timestamps and the uncongestion timestamps only for queue pairs with a more recent congestion timestamp than the uncongestion timestamp (if a queue pair has a more recent uncongestion timestamp, the queue pair is no longer considered congested). Fig.11 This process is repeated for each controller record 230 and each CoS until the device-wide record 235 is complete. Note that the weights may be reset periodically: this interval of weight reset may be any desired interval and may or may not be related to any other interval used in other calculations.

[0130] Note that when targeting Figure 2 Controller 210 Update Fig.11 When the individual controller records 230 are updated, the device range records 235 can be updated. That is, if Fig.11 The congestion timestamp in the controller record 230 is updated to a more recent time, and / or Fig.11 If the weighted congestion score in the controller record 230 of the controller is updated, this information can also be used to update the device-wide record 235. Similarly, if the queue pair receives an uncongested packet and updates the uncongested timestamp to be more recent than the congested timestamp, this information can be used to update the device-wide record 235 (because the queue pair that is no longer congested may have provided a weighted congestion score for that CoS in the controller and device-wide record 235).

[0131] Once the device range record 235 is updated, and Fig. 9 The controller association matrix 220 and Fig.10 Once the system-level association matrix 225 is prepared (this should happen during the connection phase and therefore should happen before the device range record 235 is updated), a particular controller can use all of this data to determine whether to actively apply congestion control. Specifically, for Figure 2 The specific controller 210 corresponds to Figure 2 The specific controller 210 Fig. 9 The controller association matrix 220 and Fig.10 The rows in the system-level association matrix 225 may be multiplied by a vector of weighted congestion scores from the device-wide record 235 for a particular CoS (which may factor in how recently the congestion notification was received, so some non-zero congestion scores may be omitted from the device-wide record 235).

[0132] Mathematically, given a particular priority of interest, the columns from the device-wide record 235 containing the weighted congestion scores may be extracted and formed into a column vector (which may omit timestamp information), which may be referred to as V p, where p is the CoS of interest. Then, if M C [Ci] can mean from Figure 2 of a specific controller 210, Fig. 9 The rows of the controller association matrix 220, and if M Sys [Ci] can mean from Figure 2 of a specific controller 210, Fig.10 The system-level correlation matrix of M is 225 rows. C [C i ]×V p and M Sys [C i ]×V p It can be respectively expressed for this specific CoS Figure 2 The estimated congestion score for a particular controller 210 is . Vector multiplication involves multiplying corresponding values ​​and summing the products. Mathematically, this can be expressed as

[0133] These estimated congestion scores may be compared to thresholds (each of these estimated congestion scores may be compared to the same threshold or to different thresholds): if the estimated congestion score is above the appropriate threshold, then Figure 2 The controller 210 can actively apply congestion control. Any threshold(s) may be used as desired: in some embodiments, a lower threshold may reflect a greater concern for congestion, and a higher threshold may reflect a lesser concern for congestion. For example, threshold 5 may reflect a greater tolerance for congestion than threshold 2. In some embodiments of the present disclosure, each controller may have its own threshold(s); in other embodiments of the present disclosure, thresholds may be used to determine whether to Figure 2 All controllers 210 apply active congestion control.

[0134] Note that in the above discussion, CoS is for V p Factors for selecting the value. By considering CoS as a factor, Figure 2 A particular controller 210 may manage queue pairs associated with that controller differently depending on the CoS of each queue pair. That is, some queue pairs may have active congestion control; other queue pairs may not have active congestion control. However, in some embodiments of the present disclosure, a device-wide record 235 may include a single weighted congestion score (and congestion timestamp) that applies to all priorities: Figure 2 The specific controller 210 may then apply active congestion control to Figure 2 All queue pairs associated with a particular controller 210, regardless of CoS.

[0135] In the above discussion, the congestion timestamp stored in the device-wide record 235 is used to determine whether a particular congestion notification is recent enough to be considered part of the weighted congestion score. Figure 1 The device 105 may ignore how recent a particular congestion notification is, and may use the congestion score regardless of whether the congestion notification is recent. In these embodiments of the present disclosure, old congestion notifications may be eliminated from the weighted congestion score when the congestion score itself is cleared.

[0136] Figures 13A-13B The embodiment according to the present disclosure is shown Figure 1 A flowchart of an example high-level overview of how the device 105 may apply active congestion control. Fig.13A In block 1305, for CoS, Figure 1 The device 105 may determine whether it has received a packet marked with a congestion notification. If not, then in block 1310, the packet for the particular queue pair may be sent to the transport layer without active congestion control. Otherwise, in block 1315, Figure 1 The device 105 can determine whether a packet marked with a congestion notification is received for the queue pair. If so, the queue pair has been congested controlled (at the target's transport layer), and processing can continue at block 1310.

[0137] Otherwise, at block 1320 ( Fig. 13B )middle, Figure 1 The device 105 can determine whether the queue pair has received a packet without a congestion notification more recently than the timestamp of the packet with the congestion notification. If so, the data of the queue pair can be delivered without congestion control, and processing can be performed at Fig.13A Block 1310 continues.

[0138] Otherwise, in block 1325, Figure 1 The device 105 may determine whether the packet marked with the congestion notification is for the same CoS as the queue pair, and in block 1330, Figure 1 The device 105 may determine whether the associated controller has received a packet marked with a congestion notification for the same CoS. If both boxes are negatively acknowledged, the process may proceed to Fig.13A However, if any of these blocks are acknowledged, then block 1335 is reached to check whether the congestion score of the queue pair exceeds the threshold. If the congestion score does not exceed the threshold, then processing can continue at Fig.13A Continue in block 1310; otherwise, in block 1340, Fig.13ABlock 1310 continues processing by applying active congestion control to the queue pair.

[0139] Fig.14 The embodiment according to the present disclosure is shown Figure 1 An alternative flow chart of an example process for applying active congestion control to the device 105 of FIG. Fig.14 In block 1405, Figure 1 The device 105 can identify Figure 2 The two controllers 210, and in block 1410, Figure 1 The device 105 can Figure 2 Blocks 1405 and 1410 may be performed for as many pairs of controllers 210 as are to be associated. Figure 2 The controller 210 repeats, as shown by dashed line 1415. To this end, blocks 1405 and 1410 may represent Figure 8 connection phase.

[0140] Once the connection phase is complete (at least until a new connection is established), the transfer phase may begin at block 1420, where Figure 1 Device 105 may receive packets with congestion notifications. Figure 1 If the device 105 receives a packet with a congestion notification, then in block 1425, for other controllers, Figure 2 The devices 105 may combine (eg, add) the weighted congestion scores of the associated controllers. As described above, block 1425 may include calculating M C [C i ]×V p and M Sys [C i ]×V p In block 1430, these calculated congestion scores may then be compared to one or more thresholds. If any congestion score exceeds the threshold(s), then in block 1435, Figure 2 The throttle 245 can actively apply congestion control.

[0141] Note that the above discussion does not address what happens when the connection is closed: for example, if Figure 5 The NVMe-oF initiator 105-1 closes the Figure 5 In some embodiments of the present disclosure, the closing of the connection is irrelevant: even though the queue pair may be considered to be subject to active congestion control, because the connection is closed, no data will be sent, and there is no impact on congestion regardless of whether active congestion control is applied. In other embodiments of the present disclosure, when the connection is closed, information about the controller can be found in Figure 2 The controller association matrix 220, Figure 2System-level correlation matrix 225, Figure 2 The controller records 230 and Figure 2 Specifically, for Figure 2 The controller 210, Figure 2 The controller association matrix 220 and Figure 2 The associations in the system-level association matrix 225 of can be updated to potentially remove some associations. Similarly, Figure 2 The data of the queue pair in controller record 230 can be erased, which can also trigger Figure 2 The data in the device range record 235 is updated.

[0142] Figures 15A-15B The embodiment according to the present disclosure is shown Figure 1 The device 105 determines Figure 1 Equipment 105 Figure 2 A flowchart of an example process for associating two controllers 210. Fig.15A In block 1505, Figure 1 The device 105 may determine Figure 2 Are the two controllers 210 communicating with Figure 1 Network 115 Figure 1 If yes, then at block 1510, Figure 2 The controller association matrix 220 may be updated to reflect Figure 2 Otherwise, at block 1515, Figure 1 The device 105 may determine Figure 2 Are the two controllers 210 Figure 1 Equipment 105 Figure 2 If yes, then in block 1510, Figure 2 The controller association matrix 220 may be updated to reflect Figure 2 Note that, as shown by dashed line 1520, box 1515 may be omitted.

[0143] Otherwise, at block 1525 ( Fig. 15B )middle, Figure 1 The device 105 may determine Figure 2 are communicating with devices in the same rack or cluster (that is, Figure 2 can the two controllers 210 share any switches in their communication paths). If so, then in block 1530, the Figure 2 The system-level correlation matrix 225 reflects Figure 2The two controllers 210 of FIG. 1530 are associated. Note that the association may be a fixed value (such as 1 to reflect that the controllers are associated, or 0 to reflect that the controllers are not associated), or may be within a range of values ​​(such as between 0 and 1) to reflect varying degrees of association. Otherwise, in block 1535, Figure 2 The two controllers 210 are not associated.

[0144] Note that Figures 15A-15B In the case where two controllers are Figure 2 The controller association matrix 220 of is associated with each other, then these two controllers are Figure 2 The possibility of being associated in the system-level association matrix 225 can be ignored. In some embodiments of the present disclosure, two controllers can be considered to be in Figure 2 The controller association matrix 220 and Figure 2 The system-level correlation matrix 225 of the two is related: that is, in Fig.15A After the completion of block 1510, Fig. 15B Block 1525 (as well as blocks 1530 and 1535) may also be performed.

[0145] Fig.16 The embodiment according to the present disclosure is shown Figure 1 The device 105 determines Figure 1 The two controllers of the device 105 have Figure 1 A flowchart of an example process associated with a shared switch 120. Fig.16 In block 1605, Figure 2 The path tracker 240 can identify the path from Figure 2 The controller 210 to Figure 2 The controller 210 communicates with the corresponding device's path switch. In block 1610, Figure 1 The device 105 may determine Figure 2 Controllers 210 have any common switches. If yes, then in block 1615, Figure 2 The controller 210 is considered associated; otherwise, in block 1620, Figure 2 The controller 210 is not considered associated.

[0146] Fig.17 The embodiment according to the present disclosure is shown Figure 1 A flowchart of an example process for device 105 to process congestion notifications. Fig.17 In block 1705, Figure 2 The controller 210 may receive a Figure 7 Congestion Notification Echo 720 Figure 7 In block 1710, Figure 1Device 105 can be updated Figure 2 The controller records 230 Figure 2 The controller score of the controller 210 of FIG. 1715 is Figure 1 Device 105 can be updated Figure 2 The controller records 230 Figure 2 In block 1720, Figure 1 Device 105 can be updated Figure 2 The controller records 230 Figure 2 The CoS of the queue pair of the controller 210.

[0147] In block 1725, Figure 1 The device 105 can be based on Figure 2 The controller records 230 updates to update Figure 2 Device scope record 235. For example, Figure 1 The device 105 can be based on Figure 2 The controller score and / or congestion timestamp in the controller record 230 is updated to update Figure 2 The controller score and / or congestion timestamp in the device-wide record 235 .

[0148] In block 1730, Figure 1 Device 105 receives data not marked with Figure 7 The congestion notification echo 720 packet. In block 1735, Figure 1 Device 105 can be updated Figure 2 The controller records 230 Figure 2 The uncongested timestamp of the controller 210.

[0149] Fig.18 The embodiment according to the present disclosure is shown Figure 1 After receiving the congestion notification, the device 105 determines Figure 2 Flow chart of an example process for determining a congestion score of the controller 210. Fig.18 In block 1805, the congestion score may be set to the number of congestion notifications received. The number of congestion notifications may be set during an interval, or during Figure 1 Alternatively, in block 1810, the congestion score may be set to a constant. Either way, in block 1815, the congestion score may be weighted: for example, by a function of the number of congestion notifications received (again, in Figure 1 Block 1815 may be omitted, as indicated by dashed line 1820.

[0150] In block 1825, Figure 1The device 105 can determine whether the interval has elapsed, proving Figure 2 If so, then at block 1830, the controller record 230 may be reset. As described above, upon completion of the interval, the reset may be just Figure 2 The specific queue pair in the controller record 230, Figure 2 All of the controller records 230 (but only for one controller), or Figure 2 The controller record 230 of all controllers 210. In addition, as described above, Figure 2 The controller record 230 of a controller may be reset by setting all information in the controller record 230 to zero, or, if a congestion notification was received in an increment before the end of the interval, the congestion score of the controller may be reset to a non-zero value to reflect the most recent congestion notification.

[0151] Fig.19 The embodiment according to the present disclosure is shown Figure 1 A flowchart of an example process for actively applying congestion control to device 105. Fig.19 In block 1905, Figure 2 The throttle 245 of may limit the size of the packet to the maximum segment size (for TCP packets: other limits may be used for other protocols, and typically the packet may not exceed the MTU size of the network interface card). Alternatively, in block 1910, Figure 2 The throttle 245 can limit the frequency at which packets are sent (if two or more packets are ready to be sent, the inter-packet delay is appropriately increased). By adding packets at a lower frequency, the number of packets transmitted through the network can be reduced for a period of time during which congestion can be reduced or eliminated. Note that both blocks 1905 and 1910 can apply.

[0152] In box 1915, Figure 1 The device 105 may determine whether the period during which active congestion control should be applied has ended. If so, in block 1920, the throttle 245 may stop active congestion control.

[0153] exist Figure 13A-19 In the flowchart, some embodiments of the present disclosure are shown. However, those skilled in the art will recognize that other embodiments of the present disclosure are possible by changing the order of the boxes, omitting boxes, or by including links not shown in the drawings. All these changes to the flowchart are considered embodiments of the present disclosure, whether explicitly described or not.

[0154] Embodiments of the present disclosure provide technical advantages over the prior art. By identifying congestion on associated controllers, the device can actively apply congestion control to the controllers. This active congestion control can prevent congestion affecting one controller from also affecting other controllers.

[0155] The following discussion is intended to provide a brief summary description of one or more suitable machines in which certain aspects of the present disclosure can be implemented. One or more machines can be controlled at least in part by input from traditional input devices (e.g., keyboards, mice, etc.), as well as by instructions received from another machine, interaction with a virtual reality (VR) environment, biofeedback, or other input signals. As used herein, the term "machine" is intended to broadly include a single machine, a virtual machine, or a system of communicatively coupled machines, virtual machines, or devices operating together. Exemplary machines include computing devices (such as personal computers, workstations, servers, portable computers, handheld devices, phones, tablet computers, etc.), and transportation equipment (such as private or public transportation, such as cars, trains, taxis, etc.).

[0156] One or more machines may include embedded controllers, such as programmable or non-programmable logic devices or arrays, application specific integrated circuits (ASICs), embedded computers, smart cards, etc. One or more machines may utilize one or more connections to one or more remote machines, such as through a network interface, modem, or other communication coupling. The machines may be interconnected through physical and / or logical networks, such as intranets, the Internet, local area networks, wide area networks, etc. Those skilled in the art will appreciate that network communications may utilize a variety of wired and / or wireless short-range or long-range carriers and protocols, including radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 802.11, Light, infrared, cable, laser, etc.

[0157] Embodiments of the present disclosure may be described by reference to or in conjunction with associated data including functions, procedures, data structures, applications, etc., which, when accessed by a machine, causes the machine to perform a task or define an abstract data type or a low-level hardware context. The associated data may be stored, for example, in volatile and / or non-volatile memory, such as RAM, ROM, etc., or in other storage devices and their associated storage media (including hard disks, floppy disks, optical storage, tapes, flash memory, memory sticks, digital video disks, bio-storage, etc.). The associated data may be transmitted in a transmission environment including a physical and / or logical network in the form of packets, serial data, parallel data, propagated signals, etc., and may be used in a compressed or encrypted format. The associated data may be used in a distributed environment and stored locally and / or remotely for machine access.

[0158] Embodiments of the present disclosure may include tangible, non-transitory machine-readable media including instructions executable by one or more processors, including instructions for performing elements of the present disclosure described herein.

[0159] The various operations of the above methods may be performed by any suitable means (such as various (multiple) hardware and / or software components, circuits and / or (multiple) modules) capable of performing these operations. The software may include an ordered list of executable instructions for implementing logical functions, and may be embodied in any "processor-readable medium" for use by an instruction execution system, device or apparatus (such as a single-core or multi-core processor or a system containing a processor) or in combination with an instruction execution system, device or apparatus (such as a single-core or multi-core processor or a system containing a processor).

[0160] The method or algorithm described in conjunction with the embodiments disclosed herein and the blocks or steps of the functions may be directly embodied in hardware, a software module executed by a processor, or a combination of the two. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a tangible, non-temporary computer-readable medium. The software module may reside in a random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, CD ROMs, or any other form of storage medium known in the art.

[0161] The principles of the present disclosure have been described and illustrated with reference to the illustrated embodiments, and it should be appreciated that the illustrated embodiments may be modified in arrangement and detail and may be combined in any desired manner without departing from these principles. Furthermore, although the foregoing discussion has focused on specific embodiments, other configurations may also be considered. In particular, even if expressions such as "according to embodiments of the present disclosure" are used herein, these phrases are meant to generally refer to the possibilities of the embodiments and are not intended to limit the present disclosure to specific embodiment configurations. As used herein, these terms may refer to the same or different embodiments that may be combined into other embodiments.

[0162] The foregoing illustrative embodiments should not be construed as limiting the disclosure thereof. Although several embodiments have been described, it will be readily appreciated by those skilled in the art that many modifications to these embodiments are possible without departing substantially from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined by the claims.

[0163] The embodiments of the present disclosure may be extended to the following statements, but are not limited thereto:

[0164] Statement 1. An embodiment of the present disclosure includes a device, comprising:

[0165] Network port, connected to the network;

[0166] a first controller configured to send and receive a first communication over a network using the network port;

[0167] a first storage device for a controller record of a first controller, the controller record comprising at least a first congestion score, a first congestion timestamp, and a non-congestion timestamp;

[0168] a second storage device for a device-wide record, the device-wide record comprising at least a second congestion score and a second congestion timestamp for the first controller and a third congestion score and a third congestion timestamp for the second controller, the device-wide record being based at least in part on the controller record; and

[0169] A throttle is configured to limit second communications of a second controller based at least in part on the device-wide record.

[0170] Statement 2. An embodiment of the present disclosure includes a device according to statement 1, wherein the device-wide record further includes a first class of service (CoS) for the first controller and a second CoS for the second controller.

[0171] Statement 3. An embodiment of the present disclosure includes a device according to statement 1, wherein the network port includes an Ethernet port.

[0172] Statement 4. An embodiment of the present disclosure includes an apparatus according to statement 1, wherein:

[0173] The first controller includes a first Non-Volatile Memory Express (NVMe) over Fabric (NVMe-oF) controller; and

[0174] The second controller includes a second NVMe-oF controller.

[0175] Statement 5. An embodiment of the present disclosure includes an apparatus according to statement 4, wherein:

[0176] The apparatus further comprises a third storage means for a controller association matrix; and

[0177] The throttle is configured to limit second communications of the second NVMe-oF controller based at least in part on the controller association matrix and the device range record.

[0178] Statement 6. An embodiment of the present disclosure includes an apparatus according to statement 5, wherein:

[0179] The first NVMe-oF controller is configured to send and receive first communications to the second device over the network using the network port;

[0180] The second NVMe-oF controller is configured to send and receive second communications to the second device over the network; and

[0181] The controller association matrix indicates that the first NVMe-oF controller is associated with the second NVMe-oF controller.

[0182] Statement 7. An embodiment of the present disclosure includes a device according to statement 6, wherein the second NVMe-oF controller is configured to send and receive the second communication to the second device over the network using one of the network port and the second network port.

[0183] Statement 8. An embodiment of the present disclosure includes an apparatus according to statement 5, wherein:

[0184] The first NVMe-oF controller is configured to send and receive first communications to the second device over the network using the network port;

[0185] The second NVMe-oF controller is configured to send and receive second communications to the third device over the network using the network port; and

[0186] The controller association matrix indicates that the first NVMe-oF controller is associated with the second NVMe-oF controller.

[0187] Statement 9. An embodiment of the present disclosure includes an apparatus according to statement 4, wherein:

[0188] The device also includes:

[0189] A second network port for connecting to a network;

[0190] a fourth storage device for a system level correlation matrix; and

[0191] Path tracer;

[0192] The first NVMe-oF controller is configured to send and receive first communications to the second device over the network using the network port;

[0193] The second NVMe-oF controller is configured to send and receive second communications to a third device over the network using the second network port;

[0194] The path tracker is configured to identify switches along a first path from the first NVMe-oF controller to the second device, and to identify switches along a second path from the second NVMe-oF controller to the third device;

[0195] The throttle is configured to limit second communications of the second NVMe-oF controller based at least in part on the system-level affinity matrix and the device-scope record; and

[0196] The system-level association matrix indicates that the first NVMe-oF controller is associated with the second NVMe-oF controller.

[0197] Statement 10. An embodiment of the present disclosure includes the apparatus of statement 4, wherein the throttle is configured to determine that the first communication is congested based at least in part on packets of the first communication marked with a congestion notification.

[0198] Statement 11. An embodiment of the present disclosure includes the apparatus of statement 4, wherein the throttle is configured to limit the second communication of the second controller based at least in part on the weighted congestion score and the threshold.

[0199] Statement 12. An embodiment of the present disclosure includes the device of statement 11, wherein the throttle includes a calculator for calculating a weighted congestion score based at least in part on at least one of a device-wide record, a controller affinity matrix, or a system-level affinity matrix.

[0200] Statement 13. An embodiment of the present disclosure includes the apparatus of statement 4, wherein the throttling is configured to limit at least one of a packet size of the second communication or a frequency of packets sent for the second communication.

[0201] Statement 14. An embodiment of the present disclosure includes the apparatus of statement 4, wherein the throttle is configured to limit the second communication of the second controller for a period of time.

[0202] Statement 15. An embodiment of the present disclosure comprises the apparatus of statement 14, wherein the time period comprises an estimated transmission time.

[0203] Statement 16. An embodiment of the present disclosure includes a device according to statement 4, wherein the device is at least one of an NVMe-oF initiator and an NVMe-oF target.

[0204] Statement 17. An embodiment of the present disclosure includes a method comprising:

[0205] identifying a first controller in the device and a second controller in the device;

[0206] Associating the first controller with the second controller;

[0207] determining that a first communication using a first controller is experiencing network congestion at a switch; and

[0208] Congestion control is performed on a second communication application device using a second controller.

[0209] Statement 18. An embodiment of the present disclosure includes a method according to statement 17, wherein:

[0210] The first controller includes a first Non-Volatile Memory Express (NVMe) over Fabric (NVMe-oF) controller; and

[0211] The second controller includes a second NVMe-oF controller.

[0212] Statement 19. An embodiment of the present disclosure includes a method according to statement 18, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on a first NVMe-oF controller in a device that communicates with the second device and a second NVMe-oF controller in a device that communicates with the second device.

[0213] Statement 20. An embodiment of the present disclosure includes a method according to statement 18, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on the first NVMe-oF controller communicating using a port of the device and the second NVMe-oF controller communicating using a port of the device.

[0214] Statement 21. An embodiment of the present disclosure includes the method of statement 20, wherein associating the first controller and the second controller includes marking the first NVMe-oF controller and the second NVMe-oF controller as associated in a controller association matrix.

[0215] Statement 22. An embodiment of the present disclosure includes a method according to statement 18, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on a first NVMe-oF controller in a device that communicates with the second device and a second NVMe-oF controller in a device that communicates with a third device, the second device and the third device being in a rack.

[0216] Statement 23. An embodiment of the present disclosure includes the method according to statement 22, wherein associating the first controller and the second controller further comprises:

[0217] identifying a switch along a first path from the first NVMe-oF controller to the second device; and

[0218] A switch along a second path from the second NVMe-oF controller to the third device is identified.

[0219] Statement 24. An embodiment of the present disclosure includes the method of statement 22, wherein associating the first controller and the second controller includes marking the first NVMe-oF controller and the second NVMe-oF controller as associated in a system-level association matrix.

[0220] Statement 25. An embodiment of the present disclosure includes a method according to statement 18, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on a first NVMe-oF controller in a device that communicates with the second device and a second NVMe-oF controller in a device that communicates with a third device, the second device and the third device being in a cluster.

[0221] Statement 26. An embodiment of the present disclosure includes the method according to statement 25, wherein associating the first controller and the second controller further comprises:

[0222] identifying a switch along a first path from the first NVMe-oF controller to the second device; and

[0223] A switch along a second path from the second NVMe-oF controller to the third device is identified.

[0224] Statement 27. An embodiment of the present disclosure includes the method of statement 25, wherein associating the first controller and the second controller includes marking the first NVMe-oF controller and the second NVMe-oF controller as associated in a system-level association matrix.

[0225] Statement 28. An embodiment of the present disclosure includes a method according to statement 18, wherein determining that a first communication using a first controller is subject to network congestion at a switch includes receiving a packet of the first communication marked with a congestion notification at the first NVMe-oF controller.

[0226] Statement 29. An embodiment of the present disclosure includes a method according to statement 28, wherein receiving a packet of the first communication marked with a congestion notification at the first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with a congestion timestamp of the packet.

[0227] Statement 30. An embodiment of the present disclosure includes the method of statement 28, wherein receiving the congestion notification-tagged packet of the first communication at the first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with a congestion score.

[0228] Statement 31. An embodiment of the present disclosure includes a method according to statement 30, wherein the congestion score comprises a count of a number of congestion notifications received at the first NVMe-oF controller.

[0229] Statement 32. An embodiment of the present disclosure includes a method according to statement 31, wherein the congestion score comprises a count of the number of congestion notifications received at the first NVMe-oF controller during the interval.

[0230] Statement 33. An embodiment of the present disclosure includes the method of statement 30, wherein the congestion score comprises a constant value.

[0231] Statement 34. An embodiment of the present disclosure includes a method according to statement 30, wherein updating a controller record of the first NVMe-oF controller with a congestion score includes weighting the congestion score of the first NVMe-oF controller using a weight to produce a weighted congestion score.

[0232] Statement 35. An embodiment of the present disclosure includes a method according to statement 34, wherein the weight comprises a number of congestion notifications received at the first NVMe-oF controller.

[0233] Statement 36. An embodiment of the present disclosure includes a method according to statement 35, wherein the weight comprises a number of congestion notifications received at the first NVMe-oF controller during the interval.

[0234] Statement 37. An embodiment of the present disclosure includes a method according to statement 35, wherein the weight comprises a reciprocal of a number of congestion notifications received at the first NVMe-oF controller.

[0235] Statement 38. An embodiment of the present disclosure includes a method according to statement 30, wherein receiving a packet of a first communication marked with a congestion notification at the first NVMe-oF controller further comprises updating a device-wide record with a congestion score of the first NVMe-oF controller.

[0236] Statement 39. An embodiment of the present disclosure includes a method according to statement 28, wherein receiving a packet of the first communication marked with a congestion notification at the first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with an uncongested timestamp of a last uncongested packet.

[0237] Statement 40. An embodiment of the present disclosure includes a method according to statement 28, wherein receiving a packet of a first communication marked with a congestion notification at a first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with a class of service (CoS) of a queue pair associated with the packet.

[0238] Statement 41. An embodiment of the present disclosure includes the method of statement 18, wherein determining that the first communication using the first controller is subject to network congestion at the switch further comprises comparing a congestion score of the second NVMe-oF controller to a threshold.

[0239] Statement 42. An embodiment of the present disclosure includes a method according to statement 41, wherein comparing the congestion score of the second NVMe-oF controller to the threshold value includes using a weighted congestion score of the first NVMe-oF controller as the congestion score of the second NVMe-oF controller.

[0240] Statement 43. An embodiment of the present disclosure includes the method according to statement 41, wherein comparing the congestion score of the second NVMe-oF controller to a threshold value includes:

[0241] identifying a first weighted congestion score for a first NVMe-oF controller;

[0242] identifying a second weighted congestion score for a third NVMe-oF controller, the third NVMe-oF controller being associated with the second NVMe-oF controller; and

[0243] The first weighted congestion score of the first NVMe-oF controller and the second weighted congestion score of the third NVMe-oF controller are combined to generate a congestion score of the second NVMe-oF controller.

[0244] Statement 44. An embodiment of the present disclosure includes a method according to statement 41, wherein congestion control of a second communication application device using a second controller includes congestion control of a second communication application device using a second controller based at least in part on a congestion score of the second NVMe-oF controller exceeding a threshold.

[0245] Statement 45. An embodiment of the present disclosure includes a method according to statement 18, wherein congestion control for a second communication application device using the second controller includes at least one of limiting a packet size of the second communication to a maximum segment size and limiting a frequency of packets sent for the second communication.

[0246] Statement 46. An embodiment of the present disclosure includes the method of statement 18, wherein congestion controlling the second communication application device using the second controller includes congestion controlling the second communication application device for a time period.

[0247] Statement 47. An embodiment of the present disclosure includes a method according to statement 46, wherein the time period includes an estimated transmission time.

[0248] Statement 48. An embodiment of the present disclosure includes a method according to statement 18, wherein the device is at least one of an NVMe-oF initiator and an NVMe-oF target.

[0249] Statement 49. Embodiments of the present disclosure include an article of manufacture comprising a non-transitory storage medium having stored thereon instructions that, when executed by a machine, result in:

[0250] identifying a first controller in the device and a second controller in the device;

[0251] Associating the first controller with the second controller;

[0252] determining that a first communication using a first controller is experiencing network congestion at a switch; and

[0253] Congestion control is performed on a second communication application device using a second controller.

[0254] Statement 50. Embodiments of the present disclosure include an article according to statement 49, wherein:

[0255] The first controller includes a first Non-Volatile Memory Express (NVMe) over Fabric (NVMe-oF) controller; and

[0256] The second controller includes a second NVMe-oF controller.

[0257] Statement 51. Embodiments of the present disclosure include an article according to statement 50, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on the first NVMe-oF controller in the device that communicates with the second device and the second NVMe-oF controller in the device that communicates with the second device.

[0258] Statement 52. An embodiment of the present disclosure includes an article according to statement 50, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on the first NVMe-oF controller communicating using a port of the device and the second NVMe-oF controller communicating using a port of the device.

[0259] Statement 53. An embodiment of the present disclosure includes an article of manufacture according to statement 52, wherein associating the first controller and the second controller comprises marking the first NVMe-oF controller and the second NVMe-oF controller as associated in a controller association matrix.

[0260] Statement 54. An embodiment of the present disclosure includes an article according to statement 50, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on the first NVMe-oF controller in a device that communicates with the second device and the second NVMe-oF controller in a device that communicates with the third device, the second device and the third device being in a rack.

[0261] Statement 55. Embodiments of the present disclosure include an article of manufacture according to statement 54, wherein associating the first controller and the second controller further comprises:

[0262] identifying a switch along a first path from the first NVMe-oF controller to the second device; and

[0263] A switch along a second path from the second NVMe-oF controller to the third device is identified.

[0264] Statement 56. An embodiment of the present disclosure includes an article of manufacture according to statement 54, wherein associating the first controller and the second controller comprises marking the first NVMe-oF controller and the second NVMe-oF controller as associated in a system-level association matrix.

[0265] Statement 57. An embodiment of the present disclosure includes an article according to statement 50, wherein associating the first controller and the second controller includes associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on the first NVMe-oF controller in a device that communicates with the second device and the second NVMe-oF controller in a device that communicates with a third device, the third device and the second device being in a cluster.

[0266] Statement 58. Embodiments of the present disclosure include an article of manufacture according to statement 57, wherein associating the first controller and the second controller further comprises:

[0267] identifying a switch along a first path from the first NVMe-oF controller to the second device; and

[0268] A switch along a second path from the second NVMe-oF controller to the third device is identified.

[0269] Statement 59. An embodiment of the present disclosure includes an article of manufacture according to statement 57, wherein associating the first controller and the second controller comprises marking the first NVMe-oF controller and the second NVMe-oF controller as associated in a system-level association matrix.

[0270] Statement 60. An embodiment of the present disclosure includes an article of manufacture according to statement 50, wherein determining that a first communication using a first controller is subject to network congestion at a switch includes receiving, at the first NVMe-oF controller, a packet of the first communication marked with a congestion notification.

[0271] Statement 61. An embodiment of the present disclosure includes an article of manufacture according to statement 60, wherein receiving a packet of the first communication marked with a congestion notification at the first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with a congestion timestamp of the packet.

[0272] Statement 62. An embodiment of the present disclosure includes an article of manufacture according to statement 60, wherein receiving, at the first NVMe-oF controller, a packet of the first communication marked with a congestion notification includes updating a controller record of the first NVMe-oF controller with a congestion score.

[0273] Statement 63. An embodiment of the present disclosure includes an article of manufacture according to statement 62, wherein the congestion score comprises a count of a number of congestion notifications received at the first NVMe-oF controller.

[0274] Statement 64. An embodiment of the present disclosure includes an article of manufacture according to statement 63, wherein the congestion score comprises a count of a number of congestion notifications received at the first NVMe-oF controller during the interval.

[0275] Statement 65. An embodiment of the present disclosure comprises an article of manufacture according to statement 62, wherein the congestion score comprises a constant value.

[0276] Statement 66. An embodiment of the present disclosure includes an article of manufacture according to statement 62, wherein updating a controller record of the first NVMe-oF controller with a congestion score includes weighting the congestion score of the first NVMe-oF controller using a weight to produce a weighted congestion score.

[0277] Statement 67. An embodiment of the present disclosure includes an article according to statement 66, wherein the weight comprises a number of congestion notifications received at the first NVMe-oF controller.

[0278] Statement 68. An embodiment of the present disclosure includes an article of manufacture according to statement 67, wherein the weight comprises a number of congestion notifications received at the first NVMe-oF controller during the interval.

[0279] Statement 69. An embodiment of the present disclosure includes an article according to statement 67, wherein the weight comprises a reciprocal of a number of congestion notifications received at the first NVMe-oF controller.

[0280] Statement 70. An embodiment of the present disclosure includes an article of manufacture according to statement 62, wherein receiving, at the first NVMe-oF controller, a packet of the first communication marked with a congestion notification further comprises updating a device-wide record with a congestion score of the first NVMe-oF controller.

[0281] Statement 71. An embodiment of the present disclosure includes an article according to statement 60, wherein receiving a packet of a first communication marked with a congestion notification at a first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with an uncongested timestamp of a last uncongested packet.

[0282] Statement 72. An embodiment of the present disclosure includes an article according to statement 60, wherein receiving a packet of a first communication marked with a congestion notification at a first NVMe-oF controller includes updating a controller record of the first NVMe-oF controller with a class of service (CoS) of a queue pair associated with the packet.

[0283] Statement 73. An embodiment of the present disclosure includes an article according to statement 50, wherein determining that the first communication using the first controller is subject to network congestion at the switch further comprises comparing a congestion score of the second NVMe-oF controller to a threshold.

[0284] Statement 74. An embodiment of the present disclosure includes an article of manufacture according to statement 73, wherein comparing the congestion score of the second NVMe-oF controller to the threshold value includes using a weighted congestion score of the first NVMe-oF controller as the congestion score of the second NVMe-oF controller.

[0285] Statement 75. An embodiment of the present disclosure includes an article of manufacture according to statement 73, wherein comparing the congestion score of the second NVMe-oF controller to a threshold value comprises:

[0286] identifying a first weighted congestion score for a first NVMe-oF controller;

[0287] identifying a second weighted congestion score for a third NVMe-oF controller, the third NVMe-oF controller being associated with the second NVMe-oF controller; and

[0288] The first weighted congestion score of the first NVMe-oF controller and the second weighted congestion score of the third NVMe-oF controller are combined to generate a congestion score of the second NVMe-oF controller.

[0289] Statement 76. An embodiment of the present disclosure includes an article according to statement 73, wherein congestion control of a second communication application device using a second controller includes congestion control of a second communication application device using a second controller based at least in part on a congestion score of the second NVMe-oF controller exceeding a threshold.

[0290] Statement 77. An embodiment of the present disclosure includes an article of manufacture according to statement 50, wherein congestion control for a second communication application device using the second controller includes at least one of limiting a packet size of the second communication to a maximum segment size and limiting a frequency of packets sent for the second communication.

[0291] Statement 78. An embodiment of the present disclosure includes an article of manufacture according to statement 50, wherein congestion controlling the second communication application device using the second controller includes congestion controlling the second communication application device for a time period.

[0292] Statement 79. Embodiments of the present disclosure include an article of manufacture according to statement 78, wherein the time period includes an estimated transmission time.

[0293] Statement 80. An embodiment of the present disclosure includes an article of manufacture according to statement 50, wherein the device is at least one of an NVMe-oF initiator and an NVMe-oF target.

[0294] Therefore, in view of the various arrangements of the embodiments described herein, the detailed description and accompanying materials are intended to be illustrative only and should not be taken as limiting the scope of the present disclosure. Therefore, what is claimed as the disclosure is all modifications that may fall within the scope and spirit of the following claims and their equivalents.

Claims

1. A device for congestion control, comprising: Network port, connected to the network; a first controller comprising a first Non-Volatile Memory Express (NVMe-oF) fabric-based controller, the first NVMe-oF controller configured to send and receive a first communication to a second device over the network using the network port; a second controller, the second controller comprising a second NVMe-oF controller configured to send and receive a second communication to a second device over the network using the network port; a first storage device for a controller record of the first controller, the controller record comprising at least a first congestion score, a first congestion timestamp, and a non-congestion timestamp; a second storage device for a device-wide record, the device-wide record comprising at least a second congestion score and a second congestion timestamp for the first controller and a third congestion score and a third congestion timestamp for the second controller, the device-wide record being based at least in part on the controller record; and A throttle is configured to limit second communications of a second NVMe-oF controller based at least in part on the device scope record.

2. The apparatus according to claim 1, wherein: The apparatus further comprises a third storage means for a controller association matrix; and The throttle is configured to limit second communications of the second NVMe-oF controller based at least in part on the controller association matrix and the device range record.

3. The apparatus according to claim 2, wherein: The controller association matrix indicates that the first NVMe-oF controller is associated with the second NVMe-oF controller.

4. The apparatus according to claim 1, wherein: The device also includes: a second network port connected to the network; a fourth storage device for a system level correlation matrix; and Path tracer; The second controller includes a third NVMe-oF controller configured to send and receive third communications to a third device over the network using the second network port; the path tracker being configured to identify a switch along a first path from the first NVMe-oF controller to the second device, and to identify the switch along a second path from the third NVMe-oF controller to the third device; The throttle is configured to limit third communications of the third NVMe-oF controller based at least in part on the system-level affinity matrix and the device-scope record; and The system-level association matrix indicates that the first NVMe-oF controller is associated with the third NVMe-oF controller.

5. The device according to claim 1, wherein: The throttle is configured to determine that the first communication is congested based at least in part on packets of the first communication marked with congestion notifications.

6. The device according to claim 1, wherein: The throttle is configured to limit second communications of the second NVMe-oF controller based at least in part on the weighted congestion score and a threshold.

7. The device according to claim 6, wherein: The throttle includes a calculator for calculating the weighted congestion score based at least in part on at least one of the device-wide record, a controller affinity matrix, or a system-level affinity matrix.

8. The device according to claim 1, wherein: The throttle is configured to limit at least one of a packet size of the second communication or a frequency of packets sent for the second communication.

9. A method for congestion control, comprising: Identifying a first controller in a device and a second controller in the device, the first controller comprising a first Non-Volatile Memory Express (NVMe-oF) fabric-based controller, the second controller comprising a second NVMe-oF controller; storing a controller record of the first controller in the device, the controller record comprising at least a first congestion score, a first congestion timestamp, and a non-congestion timestamp; storing a device-wide record in the device, the device-wide record comprising at least a second congestion score and a second congestion time indicator for the first controller and a third congestion score and a third congestion timestamp for the second controller, the device-wide record being based at least in part on the controller record; determining that a first communication using the first controller suffers from network congestion at a switch; as well as Based at least in part on the device-wide record, congestion control is performed on a second communication application device using the second controller.

10. The method according to claim 9, wherein: Also included is associating the first NVMe-oF controller and the second NVMe-oF controller based at least in part on a first NVMe-oF controller in the device that communicates with the second device and a second NVMe-oF controller in the device that communicates with the second device.

11. The method according to claim 9, wherein: The second controller includes a third NVMe-oF controller, The method also includes associating the first NVMe-oF controller and the third NVMe-oF controller based at least in part on the first NVMe-oF controller communicating with a second device and the third NVMe-oF controller communicating with a third device, the second device and the third device being in a rack.

12. The method according to claim 11, wherein: Associating the first NVMe-oF controller and the third NVMe-oF controller based at least in part on the first NVMe-oF controller communicating with the second device and the third NVMe-oF controller communicating with the third device includes the following operations, wherein the second device and the third device are in a rack: identifying the switch along a first path from the first NVMe-oF controller to the second device; and Identifying the switch along a second path from the third NVMe-oF controller to the third device.

13. The method according to claim 9, wherein: The second controller includes a third NVMe-oF controller, The method also includes associating the first NVMe-oF controller and the third NVMe-oF controller based at least in part on the first NVMe-oF controller communicating with a second device and the third NVMe-oF controller communicating with a third device, the second device and the third device being in a cluster.

14. The method according to claim 9, wherein: Determining that a first communication using the first controller is subject to network congestion at the switch includes receiving, at the first NVMe-oF controller, a packet of the first communication marked with a congestion notification.

15. The method according to claim 9, wherein: Determining that the first communication using the first controller is subject to network congestion at the switch also includes comparing a congestion score of the second NVMe-oF controller to a threshold.

16. The method according to claim 15, wherein: Congestion control for a second communication application device using the second NVMe-oF controller includes congestion control for the second communication application device using the second NVMe-oF controller based at least in part on a congestion score of the second NVMe-oF controller exceeding the threshold.

17. A non-transitory storage medium having stored thereon instructions which, when executed by a machine, result in: identifying a first controller in a device and a second controller in the device, the first controller comprising a first Non-Volatile Memory Express (NVMe-oF) fabric-based controller and the second controller comprising a second NVMe-oF controller; storing a controller record in the device, the controller record comprising at least a first congestion score, a first congestion timestamp, and an uncongested timestamp; storing a device-wide record in the device, the device-wide record comprising at least a second congestion score and a second congestion time indicator for the first controller and a third congestion score and a third congestion timestamp for the second controller, the device-wide record being based at least in part on the controller record; determining that a first communication using the first controller suffers from network congestion at a switch; as well as Based at least in part on the device-wide record, congestion control is performed on a second communication application device using the second controller.

18. The non-transitory storage medium of claim 17, wherein the instructions, when executed by a machine, result in implementation of the method of any one of claims 9-16.

Images