Credit-based flow control for Ethernet

Abstract

A credit-based flow control for Ethernet is provided. In a network, a receiver receives a data frame from a transmitter, the data frame including a transmitter credit count indicating a total number of credits available at the transmitter. The system determines whether one or more data frames are lost based on a transmitter credit count included in consecutively received data frames. In response to one or more data frames being lost, the system calculates an amount of credits corresponding to the lost data frames. The system then determines a number of credits to be returned to the transmitter based at least on the lost data frame and credits released by the receiver, updates the receiver credit count by increasing the number of credits to be returned to the transmitter on the instant receiver credit count, and transmits a credit control frame to the transmitter, the credit control frame includes the number of credits to return and the updated receiver credit count.

Description

[0001] Government-funded research statement

[0002] This invention was carried out with government support under contract number H98230-23-C-0350 granted by the Maryland Procurement Office. The government holds certain rights to this invention.

[0003] background Technical Field

[0004] This disclosure generally relates to credit-based flow control in networks. More specifically, this disclosure relates to implementing credit-based flow control in Ethernet. Background Technology

[0005] As data speeds on Ethernet links increase, larger buffers are needed to support priority-based flow control (PFC). Credit-based flow control has already been used to manage buffer space in various proprietary protocols. The expectation is to extend credit-based flow control to Ethernet links. However, credit-based flow control typically requires accurate credit counting (e.g., via exchanging credit control frames), and Ethernet links are inherently lossy, which affects the effectiveness of flow control mechanisms. Attached Figure Description

[0006] Figure 1A An example Ethernet data frame containing credit control information is illustrated according to one aspect of this application.

[0007] Figure 1B An example Ethernet data frame containing credit control information is illustrated according to one aspect of this application.

[0008] Figure 2 An example credit control frame according to one aspect of this application is illustrated.

[0009] Figure 3 An example process for tracking credit to facilitate credit-based flow control on an Ethernet link, according to one aspect of this application, is illustrated.

[0010] Figure 4 A flowchart illustrating an example process for updating a receiver's credit count according to one aspect of this application is presented.

[0011] Figure 5 A flowchart illustrating an example process for updating transmitter credit counts according to one aspect of this application is presented.

[0012] Figure 6 An example functional block diagram of a network device according to one aspect of this application is shown.

[0013] Figure 7 A computer-readable medium facilitating credit counting of a tracking receiver according to one aspect of this application is illustrated.

[0014] In these accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0015] Priority-based flow control (PFC) has been used to manage network traffic and prevent congestion in data center networks. PFC uses a mechanism called "pause frames" to control traffic flow. When congestion occurs, network devices with PFC capabilities (such as switches or routers) can issue pause frames to temporarily suspend transmission at a specific priority level while allowing other priority levels to continue flowing, thus preventing the loss of frames for high-priority traffic.

[0016] PFC relies on thresholds to determine when to send a pause frame. For example, when buffer usage reaches the "Xoff" threshold, a pause frame is sent to stop incoming traffic, and when buffer usage drops below the "Xon" threshold, a cancel pause frame is sent to resume traffic flow. Slip space (or a slip buffer) is required to account for latency between the buffer and the sending device. Higher link speeds typically require more slip space. Furthermore, for PFC, each of the eight priority code points (PCPs) requires slip space to prevent head-of-line congestion between traffic classes. The accumulated slip space requirements can consume significant chip area on Ethernet devices.

[0017] Given that coasting space is rarely fully utilized, in some aspects of this disclosure, a shared coasting buffer can be used instead of separate coasting buffers for different traffic categories. In other aspects, a credit-based flow control mechanism can be used to manage this shared coasting buffer to provide guaranteed buffer space for each traffic category.

[0018] Credit-based flow control has become a buffer management technique in various proprietary network protocols. In a credit-based system, the transmitter consumes credits when sending data frames to the receiver, and the receiver returns credits to the transmitter after the data frame is removed from its buffer. This allows the transmitter to manage buffer space precisely, balancing buffer and bandwidth requirements across different traffic classes. However, the effectiveness of credit-based flow control systems depends on accurate credit counting, making it complex to adapt this mechanism to existing Ethernet protocols and infrastructure, as Ethernet is inherently lossy. More specifically, during data transmission over Ethernet networks, packets can sometimes be lost, dropped, or corrupted due to various factors, making it difficult to obtain accurate counts of credits consumed by the transmitter and returned by the receiver. Combining PFC with credit-based flow control presents challenges.

[0019] To overcome this challenge, in some aspects, the transmitter and receiver can track available credits and returned credits separately, and exchange credit count information to acknowledge and account for lost credits on the line. In one example, credits can be managed as a rolling count at each end, where each time a data frame is transmitted (and thus credits are consumed), the transmitter decrements its credit count to track the amount of credits available on the transmitter, and each time credits are returned to the transmitter, the receiver increments its credit count. In some aspects, credits can be tracked for each traffic category. Depending on the implementation, each credit may correspond to a predetermined data size (e.g., 32 bytes, 664 bytes, 128 bytes, etc.).

[0020] When transmitting a data frame, the transmitter may include the current credit count available on the transmitter in the transmitted data frame. The receiver may compare the transmitter credit counts in consecutively received data frames to determine whether one or more data frames (and therefore credits) have been lost during transmission. More specifically, the receiver may determine that one or more data frames have been lost if the difference between the transmitter credit counts included in consecutively received data frames is greater than the size of the received data frame (measured in credits). For example, a newly received data frame of size l credits may indicate that the transmitter credit count at the time of data frame transmission was m, and a previously received data frame may indicate that the previous transmitter credit count was n. The receiver may calculate nml. If no data frames have been lost, nml = 0. If the result is greater than zero, the receiver may determine that credits have been lost on the line and calculate the number of lost credits based on nml.

[0021] In some aspects of this application, the transmitter credit count can be included in the preamble of an Ethernet data frame. As specified in the IEEE 802.3 standard, an Ethernet frame comprises a seven-octet preamble. The preamble of a conventional Ethernet frame consists of alternating ones and zeros. To extend credit-based flow control to Ethernet, the transmitter can be configured to modify the preamble to embed credit count information. In one example, the transmitter can generate an Ethernet frame with an eight-octet preamble, and a predetermined number of bytes in the preamble (e.g., the last four bytes) can be used to represent the current credit count at the transmitter. Note that if the transmitter credit count for a certain traffic class falls below a predetermined threshold, the transmitter can refuse to transmit data frames in that traffic class to prevent that traffic class from consuming more buffer space than its fair share.

[0022] In some cases, transmitter credit counts can be included in a special header (called the credit control header) inserted between the standard Ethernet header fields and the Ethernet payload in a manner similar to applying an 802.1Q header (i.e., a Virtual LAN (VLAN) tag). The EtherType field in the Ethernet header can be used to indicate the presence of the credit control header. In some examples, the credit control header can include eight octets. When the credit count is inserted between the Ethernet header and the payload, it can be protected by the Ethernet Frame Checksum (FCS) at the end of the Ethernet frame. When the credit count is included in a preamble, it can be protected by error-correcting codes (ECC) as part of a forward error correction (FEC) scheme implemented in high-speed Ethernet links.

[0023] Figure 1A The illustration shows an example Ethernet data frame embedded with credit control information according to one aspect of this application. Figure 1A In the Ethernet data frame 100, there are credit control header field 102, media access control (MAC) destination address field 104, MAC source address field 106, EtherType field 108, payload field 110 and frame check sequence (FCS) field 112.

[0024] The credit control header field 102 may have a length of 8 octets and may include credit count information and traffic category information. The credit count information may indicate the available credit count at the transmitter when transmitting Ethernet frame 100. The traffic category information may indicate the traffic category associated with the credit count. The transmitter maintains separate credit counts for different traffic categories. Each traffic category can be assigned a predetermined number of credits. Higher priority traffic categories can be assigned more credits compared to lower priority traffic categories. In some aspects, the system may implement dynamic buffer allocation. More specifically, the transmitter can control the allocation of receiver buffer space between traffic categories. Depending on the application, the transmitter may choose to allocate all, part, or only a small portion of the available receiver buffer space to a specific traffic category. This is possible because the transmitter has the potential benefit of knowing more about the data being transmitted and how much buffer space each traffic category will require.

[0025] In one example, the credit control header field 102 may include a credit count for a specific traffic category to which the data frame belongs. In an alternative example, the credit control header field 102 may include credit counts for all traffic categories.

[0026] The MAC destination address field 104 and the MAC source address field 106 may each comprise six octets and may respectively comprise the destination address and source address of the Ethernet frame 100. The EtherType field 108 may comprise two octets and is typically used to indicate the protocol encapsulated in the payload of the frame. The length of the payload field 110 may vary (e.g., between 42 and 1500 octets). The FCS field 112 is a four-octet Cyclic Redundancy Check (CRC) that allows the detection of corrupted data throughout the frame received at the receiver side.

[0027] Figure 1B The illustration shows an example Ethernet data frame embedded with credit control information according to one aspect of this application. Figure 1B In the Ethernet data frame 120, there are a preamble field 122, a start frame delimiter (SFD) field 124, a MAC destination address field 126, a MAC source address field 128, a credit control header field 130, an EtherType field 132, a payload field 134, and an FCS field 136.

[0028] The preamble field 122 may include seven octets of alternating ones and zeros. The SFD field 124 includes one octet and ends with a one to indicate the start of the actual frame. The MAC destination address field 126 and the MAC source address field 128 are similar to the MAC destination address field 104 and the MAC source address field 106, respectively.

[0029] Credit control header field 130 is similar to credit control header field 102 and may include credit count information and traffic category information. In some aspects, credit control header field 130 may include a transmitter credit count corresponding to a traffic category. EtherType field 132 may include a predetermined value for indicating the presence of credit control header field 130 to the device receiving Ethernet frame 120 in a manner similar to VLAN tagging. Payload field 134 and FCS field 136 are similar to payload field 110 and FCS field 112, respectively.

[0030] Ethernet frames 100 and 120 are similar to standard Ethernet frames (e.g., Ethernet frames defined by the existing IEEE 802.3 standard), except that Ethernet frame 100 replaces the 7-byte preamble field and 1-byte SFD with an 8-byte Credit Control Header field (i.e., field 102), while Ethernet frame 120 includes an additional 8-byte Credit Control Header field (i.e., field 130). Network devices that know Credit Control Header fields 102 or 130 can parse Ethernet frames 100 or 120 respectively to obtain transmitter credit counts corresponding to one or more traffic categories, while devices that do not know the Credit Control Header can process Ethernet frames 100 or 120 normally by ignoring the Credit Control Header.

[0031] Lost credits correspond to lost data frames and do not occupy buffer space on the receiver. Accordingly, in response to determining that credits have been lost on the line, the receiver can return these credits to the transmitter. In some aspects of this application, the receiver can generate a credit control frame and send it to the transmitter to return credits. In other aspects, in addition to specifying the number of credits to be returned, the credit control frame may also include receiver credit counts for one or more traffic categories. In one example, in response to determining that a data frame of a specific traffic category has been lost during transmission, the receiver can generate a credit control frame and send it to the transmitter to return the number of credits corresponding to the lost data frame. The credit control frame may specify the number of credits to be returned. Furthermore, the credit control frame may also include the current receiver credit count associated with the specific traffic category (i.e., the total number of credits to be returned). In alternative examples, the credit control frame may include the current receiver credit counts for all traffic categories.

[0032] Similar to data frames that may be lost on the line, credit control frames sent from the receiver to the transmitter may also be lost during transmission. To resolve lost credit control frames, in some respects, the transmitter can compare the receiver credit counts included in consecutively received credit control frames. For example, if the current credit control frame returns r credits and indicates a receiver credit count of s, while the previous credit control frame indicated a receiver credit count of t, the number of returned credits lost on the line can be calculated based on str. If this value is non-zero, the transmitter can determine that a certain number of credits returned by the receiver have been lost on the line. Accordingly, the transmitter can update its available credits by adding this number to the returned credits in the received credit control frames. The transmitter credit count can be updated for each traffic category.

[0033] Figure 2 An example credit control frame according to one aspect of this application is illustrated. In this example, the credit control frame 200 may resemble an existing MAC control frame (e.g., a pause frame defined by the IEEE 802.3x standard or a PFC frame defined by the IEEE 802.1Qbb standard). More specifically, the credit control frame 200 may include a preamble field 202, an SFD field 204, a MAC destination address field 206, and a MAC source address field 208, which may resemble the preamble field 122, the SFD field 124, the MAC destination address field 126, and the MAC source address field 128, respectively.

[0034] Credit control frame 200 may include an EtherType field 210, which may be similar to EtherType field 132. In some examples, the value of EtherType field 210 may be 0x8808, thereby indicating that credit control frame 200 is an Ethernet flow control frame.

[0035] Similar to the MAC control frame, payload field 212 may further include multiple fields (such as two octet MAC control opcode fields 222) to indicate that the control frame is used to return credits. Note that the opcode in the pause frame is 0x0001, while the opcode used for PFC pause is 0x0101. In this example, MAC control opcode field 222 may have a different predetermined value than the MAC control opcode in the pause frame.

[0036] Payload field 212 may include a return credit field 224 specifying the number of credits to be returned to the transmitter. When returning credits corresponding to lost data frames on the line, the number of credits to be returned, as specified by credit control frame 200, may be equal to the sum of the lengths of all lost data frames. In addition to the number of credits, return credit field 224 may also include traffic category information. For example, if a lost data frame belongs to a specific traffic category, the returned credits corresponding to that lost frame also belong to that traffic category. In addition to lost data frames, the receiver may also use credit control frame 200 to return credits to the transmitter when credits are released (e.g., when a data frame is removed from the shared buffer). In this case, the number of credits specified by return credit field 224 may also include credits corresponding to the size of the removed data frame, and return credit field 224 may further include traffic category information corresponding to the removed data frame. When frames belonging to different traffic categories are removed, return credit field 224 may include credits and traffic category information corresponding to each traffic category.

[0037] exist Figure 2 In the example shown, payload field 212 may include multiple receiver credit count fields, such as receiver credit count field TC_1226, receiver credit count field TC_228, and receiver credit count field TC_N230, one for each traffic category (TC). In this example, receiver credit counts for all traffic categories are included in the credit control frame 200. In an alternative example, payload field 212 may include only the receiver credit counts for the traffic categories in which values ​​have changed. For example, if the receiver credit count for a particular traffic category has been updated since the last credit control frame was transmitted (e.g., because a frame for that particular traffic category exited the shared buffer), payload field 212 may include the updated receiver credit count for that particular traffic category. On the other hand, if the receiver credit count for a particular traffic category remains unchanged, this information will not be included in payload field 212.

[0038] The payload field 212 may also include a PAD field 232 for zero-padding the frame. The credit control frame 200 may also include an FCS field 214, which is similar to FCS field 136. In addition to using a MAC control frame, the credit control frame may, depending on the alternative aspect, use different encodings of the Ethernet preamble to include information associated with returned credits and receiver credit counts. Such a credit control frame may have a format similar to Ethernet frame 100.

[0039] In some respects, the receiver does not need to return credits immediately after they become available (e.g., due to lost data frames or frames exiting the shared buffer). The receiver can accumulate credits to be returned. In some examples, the receiver can generate and send credit control frames in response to determining that the accumulated number of credits to be returned exceeds a predetermined threshold, in order to reduce bandwidth consumption and improve efficiency. In other examples, the receiver can generate and send credit control frames periodically. In the extreme case where no credits are available for return after a predetermined interval, the receiver can send a credit control frame with zero credits to be returned, accompanied by receiver credit counts for all traffic categories, to allow the transmitter to recover from potentially lost credit control frames. In other words, there are at least two triggering conditions for credit control frames: accumulated credits to be returned and the maximum interval between consecutive credit control frames. If either triggering condition is met, at least one credit control frame will be generated and sent to the transmitter.

[0040] To implement the credit-based flow control described above, the Ethernet devices at both the transmitter and receiver ends should be able to parse received Ethernet data and control frames (e.g., frames 100, 120, and 200) to obtain credit information. In some aspects, during the link negotiation process, the transmitter and receiver can determine whether their link partners support credit-based flow control, and if so, which frame format is supported (e.g., frame 100 or 120). In other aspects, the transmitter and receiver can exchange Link Layer Discovery Protocol (LLDP) messages. If one of the link partners does not support credit-based flow control, flow control can be performed using a conventional PFC mechanism (i.e., one buffer per traffic class). Link partners can also negotiate the size of each credit during the negotiation process.

[0041] Figure 3 An example process for tracking credit to facilitate credit-based flow control over an Ethernet link, according to one aspect of this application, is illustrated. In this example, the Ethernet link includes a transmitter 302 at one end and a receiver 304 at the other end. Note that an Ethernet node may include a transmitter and a receiver for communicating with other nodes via the Ethernet link. For example, a transmitter of a local node may send data frames to a remote node, and a receiver of a local node may receive data frames from a remote node.

[0042] During operation, transmitter 302 may send an Ethernet data frame carrying the transmitter credit count to receiver 304 (operation 306). The Ethernet data frame may include a credit control header, similar to Ethernet frame 100 or Ethernet frame 120. The transmitter credit count (i.e., the amount of credit available at the transmitter) may be embedded in the credit control header. The credit control header may also include traffic category information associated with the data frame (e.g., a priority code point (PCP) value). In one example, the credit control header may include the credit count for the specific traffic category to which the data frame belongs. In an alternative example, the credit control header may include credit counts for all traffic categories. Receiver 304 may receive the data frame (operation 308).

[0043] In response to receiving a data frame, receiver 304 can determine whether one or more data frames were lost during transmission (operation 310). Because Ethernet does not guarantee delivery, frames may be lost on the line. In some respects, receiver 304 can compare the transmitter credit counts included in a newly received frame with those in a previously received frame to determine whether one or more data frames were lost. More specifically, if the difference between the transmitter credit counts is greater than the size of the newly received frame, receiver 304 can determine that one or more data frames were lost.

[0044] If one or more data frames have been lost, receiver 304 can calculate the number of lost credits (operation 312). In one example, the number of lost credits can be calculated by subtracting the size of the newly received frame from the difference between the transmitter credit counts.

[0045] Receiver 304 can then determine the amount of credit to be returned (operation 314) and update the receiver credit count accordingly (operation 316). More specifically, the receiver credit count can be updated by adding the credits to be returned to the current count. Credits to be returned can include credits lost on the line and credits released by receiver 304. In this example, credits lost on the line correspond to frames sent by transmitter 302 but not received by receiver 304, and credits released by receiver 304 correspond to frames exiting the shared buffer. In some cases, no credits are lost on the line, and receiver 304 may only need to return released credits. In some cases, no data exits the shared buffer, so no credits are released, and receiver 304 may only have lost credits (if any) to return. In other cases, receiver 304 may return both lost and released credits.

[0046] Receiver 304 can transmit a credit control frame (operation 318) specifying the number of credits to be returned. The credit control frame is used to return credits to transmitter 302. In addition to the number of credits to be returned, the credit control frame may also include the current receiver credit count. In some examples, the credit control frame may include receiver credit counts for all traffic categories. In alternative examples, the credit control frame may only include the receiver credit count for the traffic category whose value has been updated. In some aspects, receiver 304 may accumulate credits to be returned and transmit a credit control frame in response to reaching a predetermined credit or time threshold. In some aspects, if no credits are to be returned, receiver 304 may periodically send credit control frames to transmitter 402.

[0047] Transmitter 302 receives credit control frames (operation 320) and determines whether one or more credit control frames have been lost during transmission (operation 322). In some aspects, transmitter 302 can compare receiver credit counts in consecutively received credit control frames. More specifically, transmitter 302 can calculate the difference between receiver credit counts and compare the calculated difference with the return credit specified in the received credit control frames. If the calculated difference is greater than the return credit, transmitter 302 can determine that one or more credit control frames have been lost on the line.

[0048] In response to determining that one or more credit control frames have been lost, transmitter 302 can resolve the lost credit control frames(operation 324). More specifically, transmitter 302 can calculate the number of credits to be returned lost on the line by subtracting the credits to be returned from the received credit control frames from the difference between the receiver credit counts in consecutively received credit control frames. Transmitter 302 can further calculate the number of lost credits for each traffic category.

[0049] Transmitter 302 can then update the transmitter credit count (operation 326). Updating the transmitter credit count may involve adding returned credits (including credits lost on the line as determined in operation 324) to the current transmitter credit count. In some examples, transmitter 302 may update the transmitter credit count for each traffic category. Note that although in Figure 3 Not shown, but credit needs to be synchronized on the link partner before the transmitter 320 and receiver 304 exchange their credit information (e.g., during link establishment operation). Credit on the link partner can also be resynchronized based on the current link state during data transmission.

[0050] In addition to shared buffers, the disclosed credit-based flow control mechanism can also be used to manage other constrained resources. In some aspects, this credit-based mechanism can be used to manage flow tracking resources used to allocate flow channels. Examples of flow tracking resources may include flow tables, which can be implemented using Tri-State Content Addressable Memory (TCAM) or other types of matching functions. In some aspects, in addition to credit information associated with the receiver buffer space, the credit control frame may also include additional information such as payload, credit regarding the flow channel acknowledgment buffer space, link state (which can be used to start and disconnect links to synchronize credits on link partners), Simple Link Message Protocol, watchdog keep-alive field, error and error checking information, etc.

[0051] Figure 4 A flowchart illustrating an example process for updating a receiver's credit count according to one aspect of this application is presented. Figure 4 All or any part of the operations shown can be performed, for example, by a device or group of devices implementing an Ethernet flow control protocol (e.g., PFC protocol). Although Figure 4 The example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart may be performed in different orders, and may be performed simultaneously, partially simultaneously, or in combination thereof.

[0052] During operation, the receiver can receive Ethernet data frames from the transmitter (operation 402). The Ethernet data frame may include a transmitter credit count indicating the total number of credits available at the transmitter when the Ethernet frame was transmitted. The transmitter credit count may be embedded in the preamble of the Ethernet data frame or included in a credit control header inserted between the Ethernet header and the Ethernet payload. In some examples, the data frame may include transmitter credit counts for a specific traffic category. In alternative examples, the data frame may include transmitter credit counts for all traffic categories.

[0053] Upon receiving an Ethernet data frame, the receiver can determine whether one or more Ethernet data frames were lost during transmission based on the transmitter credit count included in the received Ethernet data frame and the transmitter credit count included in previously received Ethernet data frames (operation 404). In some aspects, the receiver can compare the difference between the transmitter credit counts in consecutively received data frames with the number of credits corresponding to a newly received data frame. In other aspects, credits can be counted separately for different traffic categories, and the receiver can determine the lost Ethernet frames for each traffic category.

[0054] In response to determining that one or more Ethernet frames have been lost, the receiver can calculate the credit count corresponding to the lost Ethernet frames (operation 406). In some aspects, the receiver can subtract the credit count corresponding to the newly received frame from the difference between the transmitter credit counts in consecutively received frames. This operation can be performed for each traffic category.

[0055] The receiver can then determine the number of credits to return to the transmitter (operation 408). The receiver can calculate the number of credits to return based on lost Ethernet frames (if any) and Ethernet frames exiting the shared buffer. In some respects, the credits to be returned can be the sum of credits lost on the line and credits released by the receiver. More specifically, credits lost on the line correspond to frames sent by the transmitter but not received by the receiver, and credits released by the receiver correspond to frames exiting the shared buffer. In some cases, no credits are lost on the line, and the receiver may only need to return the released credits. In some cases, no data exits the shared buffer, so no credits are released, and the receiver may only have lost credits (if any) to return. In other cases, the receiver may return both lost credits and released credits.

[0056] The receiver can then update its credit count by increasing the instantaneous receiver credit count by the number of credits to be returned to the transmitter (operation 410). Depending on the specific scenario, the receiver may return credits to the transmitter for one or more traffic categories and then update the receiver credit count for those traffic categories. If there are no lost data frames and no frames exit the shared buffer for a traffic category, then there are no credits to be returned to the transmitter for that traffic category.

[0057] The receiver can then transmit a credit control frame to the transmitter (operation 412). The credit control frame can specify the number of credits to be returned to the transmitter. Additionally, the credit control frame can include updated receiver credit counts for one or more traffic categories. In one example, the credit control frame can include receiver credit counts for all traffic categories. In an alternative example, the credit control frame can include only the receiver credit counts for the traffic category whose values ​​have been updated. In some aspects, the receiver may not return credits immediately after they are available for return. The receiver can accumulate credits to be returned and return them after a predetermined credit or time threshold is reached. When no credits are available for return, the receiver can periodically send credit control frames to allow the transmitter to recover lost credit control frames.

[0058] Figure 5 A flowchart illustrating an example process for updating transmitter credit counts according to one aspect of this application is presented. Figure 5All or any part of the operations shown can be performed, for example, by a device or group of devices implementing an Ethernet flow control protocol (e.g., PFC protocol).

[0059] During operation, the transmitter may transmit Ethernet data frames to the receiver (operation 502). The Ethernet data frames may carry transmitter credit counts for one or more traffic categories. The transmitter may receive credit control frames from the receiver (operation 504). The credit control frames specify the amount of credits to be returned and the receiver credit counts for one or more traffic categories.

[0060] The transmitter can determine whether one or more credit control frames have been lost during transmission (operation 506). In some aspects, the transmitter can calculate the difference between the receiver credit count in consecutively received credit control frames and the credit to be returned for each traffic category. If the number of credits to be returned is less than the difference, one or more credit control frames have been lost on the line.

[0061] If one or more credit control frames have been lost on the line, the transmitter can further determine the number of lost credits to be returned (operation 508). In some examples, the transmitter can subtract the credits to be returned specified in the current credit control frame from the difference between the receiver credit counts in consecutively received credit control frames.

[0062] The transmitter can then update the transmitter credit count (operation 510). If one or more lost credit control frames have been lost, the transmitter can add the lost credits to be returned and the credits to be returned in the current control frame to the immediate transmitter credit count. If no credit control frames have been lost, the transmitter can add only the credits to be returned in the current control frame to the immediate transmitter credit count.

[0063] although Figures 3 to 5 The example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart may be performed in different orders, and may be performed simultaneously, partially simultaneously, or in combination thereof.

[0064] Figure 6 The illustration shows an example functional block diagram of a network device according to one aspect of this application. Network device 600 may include any physical device that allows hardware on a computer network to communicate and interact with each other. Examples of network device 600 may include switches, routers, gateways, access points, network interface cards (NICs), etc. Figure 6 In this context, network device 600 may include multiple communication ports, such as ports 602 and 604, for communicating with peer network devices. Each port may include a transmitter and a receiver.

[0065] Network device 600 may include one or more processing resources (e.g., processing resource 606), one or more storage devices (e.g., storage device 608), and receiver credit tracking system 610.

[0066] In the examples described herein, processing resources may include, for example, one or more processors included in a single computing device or distributed across multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to fetch and execute instructions, other electronic circuitry suitable for fetching and executing instructions stored on a computer-readable storage medium, or combinations thereof. In the examples described herein, processing resources may fetch, decode, and execute instructions stored on a storage medium to perform the function described with respect to the instructions stored on the computer-readable medium. In other examples, the function described with respect to any instructions described herein may be implemented in the form of electronic circuitry, executable instructions encoded on a computer-readable medium, or a combination thereof. The computer-readable storage medium may be located in or away from the computing device executing the instructions but accessible (e.g., via a computer network) for execution. In the examples illustrated herein, a node may be implemented by one or more computer-readable storage media. Network device 600 may include more than Figure 6 The entities shown are fewer or more entities.

[0067] The receiver credit tracking system 610 may include any number of software units, hardware units, and firmware units that work together to achieve the goal of tracking the amount of credit returned from the receiver to the transmitter. In some aspects, the receiver credit tracking system 610 may include instructions that, when executed by the processing resource 606, cause the processing resource 606 to perform the methods and / or processes described in this disclosure. Specifically, the receiver credit tracking system 610 may include instructions 612 for receiving Ethernet data frames, as described above regarding... Figure 3 Operation 308 and shown Figure 4 The operation 402 shown is described herein. In some aspects, an Ethernet data frame may include a transmitter credit count indicating the total number of credits available at the transmitter when the Ethernet frame is transmitted. In other aspects, an Ethernet data frame may include transmitter credit counts for one or more traffic categories. Examples of Ethernet data frames may include... Figure 1A Data frame 100 shown Figure 1B Data frame 120 is shown in the figure.

[0068] The receiver credit tracking system 610 may include instructions 614 for determining whether one or more Ethernet data frames have been lost during transmission, as described above. Figure 3 Operations 310 and shown Figure 4 The operation 404 shown in the figure describes this. More specifically, instruction 614 can be used to calculate the difference between transmitter credit counts included in consecutively received data frames and compare this difference with the size of the received data frames.

[0069] The receiver credit tracking system 610 may include instructions 616 for calculating the number of credits corresponding to lost Ethernet frames (if any), as described above. Figure 3 Operations 312 and shown Figure 4 The operation 406 shown is described. More specifically, instruction 616 can be used to subtract the number of credits corresponding to the received data frames from the difference between transmitter credit counts included in consecutively received data frames.

[0070] The receiver credit tracking system 610 may include instructions 618 for determining the amount of credit to be returned to the transmitter, as described above. Figure 3 Operations 314 and shown Figure 4 As described in operation 408 shown. More specifically, instruction 618 can be used to determine the amount of credits released by the receiver (which may correspond to an Ethernet data frame exiting the shared buffer on the receiver), and then add the released credits to the lost credits.

[0071] The receiver credit tracking system 610 may include instructions 620 for updating the receiver credit count based on the amount of credit to be returned to the transmitter, as described above. Figure 3 Operations 316 and shown Figure 4 The operation 410 shown is described. Instruction 620 can be used to add the credit to be returned to the instant receiver credit count.

[0072] The receiver credit tracking system 610 may include instructions 622 for transmitting credit control frames to the transmitter, as described above. Figure 3 Operations 318 and shown Figure 4 As described in operation 412 shown, the credit control frame can specify the number of credits to be returned to the transmitter and their corresponding traffic categories. The credit control frame can further include an instantaneous receiver credit count for one or more traffic categories, thereby facilitating the transmitter's resolution of lost credit control frames (if any).

[0073] Network device 600 may include more than Figure 6The entities shown are fewer or more entities. For example, network device 600 may also include a transmitter credit tracking system for tracking the amount of credit available at the transmitter of network device 600. Receiver credit tracking system 610 may include more than Figure 6 The instructions shown are further instructions. For example, the receiver credit tracking system 610 may include instructions for accumulating credit to be returned before returning credit and instructions for periodically sending credit control frames when no credit is to be returned.

[0074] Figure 7 A computer-readable medium facilitating credit counting for a tracking receiver according to one aspect of this application is illustrated. CRM 700 may be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a “computer-readable storage medium” can be any electronic, magnetic, optical, or other physical storage device for containing or storing information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein may be any one or a combination of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., compact disk, DVD, etc.). Furthermore, any computer-readable storage medium described herein may be non-transitory.

[0075] The CRM 700 can store: instructions 710 for receiving Ethernet data frames, as mentioned above. Figure 3 Operation 308 and shown Figure 4 The operation 402 shown herein describes the instruction 720 for determining whether one or more Ethernet data frames have been lost during transmission, as described above regarding... Figure 3 Operations 310 and shown Figure 4 Operation 404 shown herein describes the instruction 730 for calculating the number of credits corresponding to lost Ethernet frames (if any), as described above regarding Figure 3 Operations 312 and shown Figure 4 The operation 406 shown describes the instruction 740 for determining the amount of credit to be returned to the transmitter, based at least on the lost data frames and the credits released by the receiver, as described above regarding... Figure 3 Operations 314 and shown Figure 4 Operation 408 shown herein describes the instruction 750 for updating the receiver credit count based on the amount of credit to be returned to the transmitter, as described above regarding... Figure 3 The operation 316 shown is described; and the instruction 760 for transmitting a credit control frame to the transmitter, as described above regarding... Figure 3Operations 318 and shown Figure 4 The operation 412 shown in the figure is described.

[0076] CRM 700 can include more Figure 7 The instructions shown are further instructions. For example, the CRM 700 may include instructions for accumulating credits to be returned before returning them, and instructions for periodically sending credit control frames when no credits are to be returned.

[0077] In general, aspects of this disclosure address the technical problem of extending credit-based flow control to systems implementing Ethernet protocols while preserving the Ethernet frame format. To accurately track credit at both the transmitter and receiver sides, each transmitted data frame may include transmitter credit count information for one or more traffic categories. The transmitter credit count information may be embedded in an additional credit control header between the preamble or Ethernet header and the payload. The receiver can determine whether a data frame (and therefore credit) was lost during transmission and return the lost credit to the transmitter. The receiver may send a credit control frame to the transmitter to return the credit. Each credit control frame may include the current receiver credit count for one or more traffic categories to facilitate the transmitter's resolution of lost credit control frames. The receiver may be configured to aggregate credit control frames (e.g., by accumulating the credit to be returned before returning it). The receiver may be configured to send a credit control frame when the interval between consecutive credit control frames reaches a predetermined threshold. The receiver may also be configured to periodically send credit control frames to the transmitter when no credit is available for return, allowing the transmitter to recover lost return credit on the line.

[0078] One aspect of this application provides a system and method for implementing credit-based flow control. During operation, a receiver receives data frames from a transmitter, the data frames including a transmitter credit count indicating the total number of credits available at the transmitter at the time the data frame was transmitted. The system determines whether one or more data frames have been lost during transmission based on the transmitter credit counts included in the received data frames and the transmitter credit counts included in previously received data frames. In response to determining that one or more data frames have been lost, the system calculates the amount of credits corresponding to the lost data frames. The system then determines, at least based on the lost data frames and the credits released by the receiver, the amount of credits to be returned to the transmitter, updates the receiver credit count by increasing the immediate receiver credit count by the amount of credits to be returned to the transmitter, and transmits a credit control frame to the transmitter, the credit control frame including the amount of credits to be returned and the updated receiver credit count.

[0079] In a variation of this, the transmitter receives credit control frames and updates the transmitter credit count based at least on the amount of credit to be returned.

[0080] In a variation of this approach, the system accumulates the credits to be returned. In response to the accumulated credits exceeding a predetermined threshold, the system transmits a credit control frame.

[0081] In a variation of this aspect, in response to determining that no credit needs to be returned to the transmitter within a predetermined interval, the system transmits a credit control frame to the transmitter, the credit control frame including an instantaneous receiver credit count.

[0082] In a variation of this aspect, the transmitter credit count is included in the preamble of the data frame or in a credit control header located between multiple header fields and the payload in the data frame.

[0083] In a variation of this aspect, the data frame includes an Ethernet data frame, and the credit control frame includes an Ethernet MAC control frame having a predetermined Media Access Control (MAC) control opcode.

[0084] In a variation of this aspect, the transmitter credit count and the receiver credit count are traffic category specific, and the credit control frame includes updated receiver credit counts for one or more traffic categories.

[0085] In a variation of this, the credit is used to manage buffer space or stream tracing resources on the receiver.

[0086] The methods and processes described in the Detailed Description section can be embodied in code and / or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored in the computer-readable storage medium.

[0087] The methods and processes described above may be incorporated into hardware modules or devices. Hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or code at specific times, and other programmable logic devices now known or developed in the future. When a hardware module or device is activated, it executes the methods and processes contained therein.

[0088] The foregoing description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are intended to be consistent with the maximum scope of the principles and features disclosed herein.

[0089] Furthermore, the foregoing descriptions of the various aspects have been presented solely for illustrative and descriptive purposes. These descriptions are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A method for implementing credit-based flow control, the method comprising: At the receiver, a data frame is received from the transmitter, the data frame including a transmitter credit count indicating the total number of credits available at the transmitter when the data frame was transmitted; It is determined whether one or more data frames were lost during transmission based on the transmitter credit count included in the received data frame and the transmitter credit count included in the previously received data frame. In response to determining that one or more data frames have been lost, calculate the credit count corresponding to the lost data frames; The amount of credit to be returned to the transmitter is determined based at least on the lost data frames and the credits released by the receiver; The receiver credit count is updated by increasing the amount of credit to be returned to the transmitter in the instant receiver credit count. as well as A credit control frame is transmitted to the transmitter, the credit control frame including the number of credits to be returned and the updated receiver credit count.

2. The method of claim 1, further comprising: The credit control frame is received at the transmitter. as well as The transmitter credit count is updated based at least on the number of credits to be returned.

3. The method of claim 1, further comprising: Accumulate the credit that is to be returned; as well as In response to the accumulation of credits to be returned exceeding a predetermined threshold, the credit control frame is transmitted.

4. The method of claim 1, further comprising: In response to determining that no credit needs to be returned to the transmitter within a predetermined interval, a credit control frame is transmitted to the transmitter, the credit control frame including the instantaneous receiver credit count.

5. The method of claim 1, wherein, The transmitter credit count is included in the preamble of the data frame or in the credit control header located between multiple header fields and the payload in the data frame.

6. The method of claim 1, wherein, The data frame includes an Ethernet data frame, and wherein the credit control frame includes an Ethernet MAC control frame having a predetermined Media Access Control (MAC) control opcode.

7. The method of claim 1, wherein, The transmitter credit count and the receiver credit count are traffic category specific, and the credit control frame includes the updated receiver credit count for one or more traffic categories.

8. The method of claim 1, wherein, The credit is used to manage buffer space or stream tracking resources on the receiver.

9. The method of claim 1, wherein, The credit control frame includes status information associated with the link between the transmitter and the receiver, which facilitates credit count synchronization on the transmitter and the receiver.

10. A network device, comprising: Processing resources; as well as A non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium including instructions that can be executed by the processing resource to perform the following operations: At the receiver, a data frame is received from the transmitter, the data frame including a transmitter credit count indicating the total number of credits available at the transmitter when the data frame was transmitted; It is determined whether one or more data frames were lost during transmission based on the transmitter credit count included in the received data frame and the transmitter credit count included in the previously received data frame. In response to determining that one or more data frames have been lost, calculate the credit count corresponding to the lost data frames; The amount of credit to be returned to the transmitter is determined based at least on the lost data frames and the credits released by the receiver; The receiver credit count is updated by increasing the amount of credit to be returned to the transmitter in the instant receiver credit count. as well as A credit control frame is transmitted to the transmitter, the credit control frame including the number of credits to be returned and the updated receiver credit count.

11. The network device of claim 10, wherein the instructions are further configured to perform the following operations: Accumulate the credit to be returned; and In response to the accumulation of credits to be returned exceeding a predetermined threshold, the credit control frame is transmitted.

12. The network device of claim 10, wherein the instructions are further configured to perform the following operations: In response to determining that no credit needs to be returned to the transmitter within a predetermined interval, a credit control frame is transmitted to the transmitter, the credit control frame including the instantaneous receiver credit count.

13. The network device as claimed in claim 10, wherein, The data frame includes an Ethernet data frame, wherein the transmitter credit count is included in the preamble of the Ethernet data frame or in a credit control header located between a plurality of header fields and the payload in the Ethernet data frame, and wherein the credit control frame includes an Ethernet MAC control frame having a predetermined Media Access Control (MAC) control opcode.

14. The network device as claimed in claim 10, wherein, The transmitter credit count and the receiver credit count are traffic category specific, and the credit control frame includes the updated receiver credit count for one or more traffic categories.

15. The network device as claimed in claim 10, wherein, The credit is used to manage buffer space or stream tracking resources on the receiver.

16. A non-transitory computer-readable storage medium storing instructions for performing the following operations: At the receiver, a data frame is received from the transmitter, the data frame including a transmitter credit count indicating the total number of credits available at the transmitter when the data frame was transmitted; It is determined whether one or more data frames were lost during transmission based on the transmitter credit count included in the received data frame and the transmitter credit count included in the previously received data frame. In response to determining that one or more data frames have been lost, calculate the credit count corresponding to the lost data frames; The amount of credit to be returned to the transmitter is determined based at least on the lost data frames and the credits released by the receiver; The receiver credit count is updated by increasing the amount of credit to be returned to the transmitter in the instant receiver credit count. as well as A credit control frame is transmitted to the transmitter, the credit control frame including the number of credits to be returned and the updated receiver credit count.

17. The non-transitory machine-readable storage medium of claim 16, wherein the instructions are further configured to perform the following operations: Accumulate the credit to be returned; and In response to the accumulation of credits to be returned exceeding a predetermined threshold, the credit control frame is transmitted.

18. The non-transitory machine-readable storage medium of claim 16, wherein the instructions are further configured to perform the following operations: In response to determining that no credit needs to be returned to the transmitter within a predetermined interval, a credit control frame is transmitted to the transmitter, the credit control frame including the instantaneous receiver credit count.

19. The non-transitory machine-readable storage medium as described in claim 16, in, The data frame includes an Ethernet data frame; The transmitter credit count is included in the preamble of the Ethernet data frame or in a credit control header located between multiple header fields and the payload in the Ethernet data frame; and The credit control frame includes an Ethernet MAC control frame with a predetermined Media Access Control (MAC) control opcode.

20. The non-transitory machine-readable storage medium of claim 16, wherein, The transmitter credit count and the receiver credit count are traffic category specific, and the credit control frame includes the updated receiver credit count for one or more traffic categories.

Images