Rate update engine for reliable transport protocol

Abstract

The invention relates to a rate update engine for reliable transport protocols. A system includes a first processor configured to analyze packets received over a communication protocol system and determine one or more congestion indicators from the analysis of the data packets, the one or more congestion indicators indicating network congestion of data packets transmitted over a reliable transport protocol layer of the communication protocol system. The system also includes a rate update engine separate from the packet data path and configured to operate the second processor to receive the determined one or more congestion indicators, one or more congestion control parameters for controlling transmission of the data packet are determined based on the received one or more congestion indicators and a congestion control result based on the determined one or more congestion control parameters is output.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application 202011186707.8, with a filing date of October 29, 2020. Technical Field

[0002] The present disclosure relates to a rate update engine for a reliable transport protocol. Background Art

[0003] The Internet Protocol Suite is a collection of communication protocols used to serve the transmission of data between two devices communicating information over the Internet or other computer networks. The Transmission Control Protocol ("TCP") is part of the Internet Protocol Suite that provides connection-oriented, reliable, and ordered delivery of data packet streams between a web browser application running on a client device and a web server application running on a server device, for example, over a local area network or a wide area network. Currently, data centers that use communication protocols such as TCP are experiencing certain problems. For example, incast is a many-to-one communication mode commonly found in data centers that can cause incast congestion when multiple synchronous computing devices send data to the same receiver computing device in parallel. Summary of the invention

[0004] The present disclosure provides a programmable congestion control engine that can be used to implement various congestion control routines to avoid or alleviate overcast congestion at remote nodes of a communication network, such as receiver computing devices. The congestion control engine collects information from remote nodes to monitor congestion, and can therefore update the rate of packet transmission to the remote nodes using a rate update engine (RUE). The RUE can be separated from the packet data path. This offloads the processing required to perform congestion control to hardware or software with sufficient processing power, and further allows flexibility in the type of one or more congestion control algorithms that can be used.

[0005] One aspect of the present disclosure is directed to a congestion control system, comprising one or more first processors of a first node in a communication network, wherein the one or more first processors are communicatively coupled to a packet data path of the communication network for receiving data packets using a reliable transport protocol and are configured to analyze packets received from a second node and determine one or more congestion indicators based on the analysis of the data packets, the one or more congestion indicators indicating network congestion of data packets transmitted to the second node through a reliable transport protocol layer, and a rate update engine separate from the packet data path, wherein the rate update engine is configured to operate the one or more second processors to receive the determined one or more congestion indicators, determine one or more congestion control parameters for controlling transmission of data packets through the communication network to the second node based on the received one or more congestion indicators, and output a congestion control result based on the determined one or more congestion control parameters.

[0006] In some examples, the one or more first processors may be configured to provide the determined one or more congestion indicators to the one or more second processors in response to an occurrence of a packet transmission event. The packet transmission event may be one of an acknowledgement message, a non-acknowledgement message, or a packet retransmission.

[0007] In some examples, the one or more congestion indicators may include at least one of the following for data packets transmitted to the second node: a congestion window, a retransmission timeout, a round-trip delay, a number of packets acknowledged by the second node, a number of retransmissions, or a number of explicit congestion notifications received.

[0008] In some examples, the rate update engine may include a dedicated hardware engine.The rate update engine, the one or more first processors, and the one or more second processors may be included in the same hardware component.

[0009] In some examples, the rate update engine may include a memory storing instructions. The memory may further include a first queue configured to queue input data provided from the one or more first processors, and a second queue configured to queue output data provided to the one or more first processors. The first queue may be a first circular buffer having a first head pointer and a first tail pointer. The second queue may be a second circular buffer having a second head pointer and a second tail pointer. The one or more first processors may be configured to push a rate update request to the first circular buffer according to the first tail pointer, and read a rate update request from the second circular buffer according to the second head pointer. For each rate update request, the one or more first processors may be configured to determine whether to push a rate update request to the first circular buffer based on whether the amount of time since the previous rate update request exceeds a predetermined delay threshold or whether the number of ACK packets received for the data packet transmitted to the second node exceeds a predetermined throughput threshold. The one or more first processors may be configured to determine a current number of rate update requests queued in the first circular buffer, push the rate update request to the first circular buffer if the current number of rate update requests queued in the first circular buffer is less than a first threshold amount, avoid pushing the rate update request to the first circular buffer if the current number of rate update requests queued in the first circular buffer is greater than the first threshold amount and less than a second threshold amount, the amount of time since a previous rate update request does not exceed a predetermined latency threshold, and the number of ACK packets received for data packets transmitted to the second node does not exceed a predetermined throughput threshold, and if the current number of rate update requests queued in the first circular buffer is greater than the first threshold amount and less than a second threshold amount, avoid pushing the rate update request to the first circular buffer If the current number of rate update requests queued in a circular buffer is greater than a second threshold amount and less than a third threshold amount, and the amount of time since a previous rate update request does not exceed a predetermined latency threshold, then the rate update request is avoided from being pushed to the first circular buffer, and if the current number of rate update requests queued in the first circular buffer is greater than the third threshold amount, and at least one of the following two conditions is also met: (i) the amount of time since the previous rate update request does not exceed the predetermined latency threshold, and (ii) the number of ACK packets received for the data packets transmitted to the second node does not exceed a predetermined throughput threshold, then the rate update request is avoided from being pushed to the first circular buffer.

[0010] Another aspect of the present disclosure is directed to a method, including: analyzing, by one or more processors of a first node in a communication protocol system, data packets received from a second node via a packet data path; determining, by the one or more processors, one or more congestion indicators based on the analysis of the data packets, wherein the one or more congestion indicators indicate network congestion for data packets transmitted to the second node via a reliable transport protocol layer of the communication protocol system; transmitting, by the one or more processors, the determined one or more congestion indicators to a rate update engine separated from the packet data path; receiving, by the one or more processors, a congestion control result based on the determined one or more congestion control parameters from the rate update engine; and controlling, by the one or more processors, transmission of the data packets to the second node based on the congestion control result.

[0011] In some examples, the method may further include generating, by the one or more processors, a rate update request in response to receiving an acknowledgement message, receiving a non-acknowledgement message, or retransmitting a packet. The method may further include transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to a first queue of a mailbox mechanism, and obtaining, by the one or more processors, the congestion control result from a second queue of the mailbox mechanism. The method may further include limiting the number of generated rate update requests transmitted to the first queue based at least in part on the available remaining capacity of the first queue.

[0012] In some examples, limiting the number of rate update requests generated may involve accessing, by the one or more processors, an available remaining capacity of the first queue, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue if the available remaining capacity of the first queue is less than a first threshold amount, determining, by the one or more processors, an amount of time that has elapsed since a most recent rate update if the available remaining capacity of the first queue is greater than or equal to the first threshold amount, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue if the amount of time that has elapsed since the most recent rate update is greater than a predetermined latency threshold, and avoiding, by the one or more processors, transmitting rate update requests to the first queue if the amount of time that has elapsed since the most recent rate update is less than or equal to the predetermined latency threshold, thereby limiting the number of rate update requests transmitted to the first queue.

[0013] In some examples, limiting the number of rate update requests generated may involve accessing, by the one or more processors, the available remaining capacity of the first queue, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue if the available remaining capacity of the first queue is less than a first threshold amount, and determining, by the one or more processors, the number of ACK packets received for data packets transmitted to the second node if the available remaining capacity of the first queue is greater than or equal to the first threshold amount, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue if the number of received ACK packets is greater than a predetermined throughput threshold, and avoiding transmitting, by the one or more processors, the rate update request to the first queue if the number of received ACK packets is less than or equal to the predetermined throughput threshold, thereby limiting the number of rate update requests transmitted to the first queue.

[0014] In some examples, limiting the number of rate update requests generated can involve accessing, by the one or more processors, an available remaining capacity of the first queue, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue if the available remaining capacity of the first queue is less than a first threshold amount, determining, by the one or more processors, an amount of time that has elapsed since a most recent rate update and a number of ACK packets received for data packets transmitted to the second node if the available remaining capacity of the first queue is greater than the first threshold amount and less than a second threshold amount, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue if the amount of time that has elapsed since the most recent rate update is greater than a predetermined latency threshold or the number of ACK packets received is greater than a predetermined throughput threshold, and refraining, by the one or more processors from transmitting rate update requests to the first queue if the amount of time that has elapsed since the most recent rate update is less than or equal to the predetermined latency threshold and the number of ACK packets received is less than or equal to the predetermined throughput threshold, thereby limiting the number of rate update requests transmitted to the first queue. the number of rate update requests to be sent, if the available remaining capacity of the first queue is greater than a second threshold amount and less than a third threshold amount, if the amount of time that has elapsed since the most recent rate update is greater than a predetermined latency threshold, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue, and if the amount of time that has elapsed since the most recent rate update is less than or equal to the predetermined latency threshold, refraining, by the one or more processors from transmitting a rate update request to the first queue, thereby limiting the number of rate update requests transmitted to the first queue, and if the available remaining capacity of the first queue is greater than a second threshold amount and less than a third threshold amount, transmitting, by the one or more processors, the rate update request and a congestion indicator associated with the rate update request to the first queue, If the amount of time that has elapsed since the most recent rate update is greater than a predetermined delay threshold and the number of ACK packets received is greater than a predetermined throughput threshold, the one or more processors transmit the rate update request and the congestion indicator associated with the rate update request to the first queue; and if the amount of time that has elapsed since the most recent rate update is less than or equal to the predetermined delay threshold or the number of ACK packets received is less than or equal to at least one of the predetermined throughput thresholds, the one or more processors avoid transmitting the rate update request to the first queue, thereby limiting the number of rate update requests transmitted to the first queue.

[0015] In some aspects, the method may further include receiving, by one or more second processors controlled by the rate update engine, the rate update request and a congestion indicator associated with the rate update request from the first queue, determining, by the one or more second processors, one or more control parameters for controlling transmission of data packets to the second node based on the received one or more congestion indicators, and outputting, by the one or more second processors, a congestion control result based on the determined one or more congestion control parameters to the second queue.

[0016] Yet another aspect of the present disclosure is directed to a non-transitory computer-readable medium having instructions stored thereon, the instructions being used to control one or more processors of a first node in a communication protocol system to perform the following steps: analyzing data packets received from a second node via a packet data path; determining one or more congestion indicators based on the analysis of the data packets, wherein the one or more congestion indicators indicate network congestion of data packets transmitted to the second node via a reliable transport protocol layer of the communication protocol system; transmitting the determined one or more congestion indicators to a rate update engine separate from the packet data path; receiving a congestion control result based on the determined one or more congestion control parameters from the rate update engine; and controlling transmission of the data packets to the second node based on the congestion control result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram of an example communication network according to aspects of the present disclosure.

[0018] Figure 2 is a block diagram of an example system according to aspects of the present disclosure.

[0019] Figure 3 yes Figure 2 A block diagram of an example implementation of a system.

[0020] Figure 4 yes Figure 3 Block diagram of an example computing device included in an example system of FIG.

[0021] Figure 5 yes Figure 2 A block diagram of another example implementation of a system.

[0022] Figure 6 yes Figure 5 Schematic diagram of an example memory included in an example system.

[0023] Figure 7 yes Figure 5 Graphical depiction of an example protocol executed by an example system.

[0024] Figure 8is a flow chart of an example rate update routine according to aspects of the present disclosure.

[0025] Fig. 9 is a flow chart of an example rate update request subroutine according to aspects of the present disclosure. DETAILED DESCRIPTION Overview

[0026] The technology generally relates to implementing congestion control for a reliable transport protocol (RTP) layer of a communication protocol system separately from the main data path of the RTP layer. The main data path may include one or more first processors for analyzing incoming packets and determining when to initiate a rate update event. The first processor may send a rate update request to a rate update engine, which controls one or more second processors located outside the main data path. The second processor may execute a rate update algorithm to determine a control parameter for congestion control. The second processor may then provide the control parameter back to the first processor to implement congestion control on the main data path.

[0027] The rate update request may be triggered by hardware events such as an ACK packet, a NACK packet, or a packet retransmission. Any of these events may indicate a change in congestion conditions at a remote node receiving data packets via the RTP layer, which in turn may warrant an update to congestion control at the transmitting node.

[0028] The execution of the rate update algorithm can be implemented by dedicated hardware or by a software module outside the main data path. In the example of dedicated hardware, the second processor can be included in a dedicated hardware engine as a part of hardware for implementing RTP in a communication protocol system, which for example includes data path node hardware.

[0029] In an example of a software module, the software module may include instructions for implementing a congestion control algorithm at the one or more second processors.A pair of ring buffers may be used to queue data to and from the rate update scheduler of the primary data path and the offline software-based rate update engine.

[0030] In an example where a circular buffer is used to queue data sent from a primary data path to an offline rate update engine, a rate limiter may be used to control the number of rate update requests that are pushed to the offline rate update engine. Limiting the number of rate update requests may be particularly advantageous in those examples where the rate update engine is implemented in software, because software may be more limited than dedicated hardware in the number of congestion control events that can be handled. The rate limiter may analyze the remaining capacity of the circular buffer and the importance of increasing the rate update request, and may determine whether to push the rate update request to the circular buffer based on this information. The importance of the rate update request may be measured based on the latency of the event, the throughput of the transmitted data packets, or a combination thereof.

[0031] The above implementation can offload processing steps from the main data path to a separate offline location while also providing robust congestion control. Furthermore, the offline implementation of congestion control is flexible in that the proposed architecture is not limited to any particular type of congestion control or rate update algorithm. Furthermore, bursts and temporary reverse pressures between the main data path and the offline engine can be absorbed using a rate limiter to ensure that the main data path does not stall. Example System

[0032] Figure 1 An example communication network 100 is illustrated that includes one or more nodes. The nodes may represent various computing devices that are interconnected via network connections. Figure 1 In the example of , the first node 110 and the second node 120 are shown to be connected via a network 130 .

[0033] Each of the nodes 110, 120 may include a respective processor 112, 122, memory 114, 124, and a communication device 116, 126 for receiving input and transmitting output.

[0034] The processors 112, 122 may be known processors or other less well-known processor types. Alternatively, the processors 112, 122 may be dedicated controllers such as ASICs.

[0035] The memory 114, 124 may store information accessible by the processors 112, 122, including data that can be retrieved, manipulated, or stored by the processors, instructions that can be executed by the processors, or a combination thereof. The memory 114, 124 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processors 110, 120, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writeable and read-only memory.

[0036] Although Figure 1 The processor 112, 122 and corresponding memory 114, 124 of each node are functionally illustrated as being included in a single box, but the processor and memory may actually include multiple processors and memories that may or may not be stored in the same physical housing. For example, some data and instructions may be stored on a removable CD-ROM, while others are stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location that is physically far away from the processor but still accessible to it. Similarly, the processor may actually include a collection of processors that may or may not operate in parallel.

[0037] The communication equipment 116, 126 of each node of the communication network can facilitate the communication between the node and other remote devices communicating with the node. The remote device may include other nodes of the communication network 100, one or more user devices communicating with the node, or any combination thereof. The communication equipment 116, 126 may be able to communicate with other computers such as modems (e.g., dial-up, cable or optical fiber) and wireless interfaces to transmit data. For example, each node can receive communication via a network connection 130 (such as a private network, Ethernet, WiFi (e.g., 702.71, 702.71b, g, n or other such standards), and RPC, HTTP, and various combinations thereof).

[0038] The network connections 130 between the nodes 110, 120 of the communication network 100 may support multiple levels of communication protocols implemented by the nodes' respective processors 112, 122. For example, each node may support communications using an upper layer protocol (ULP) and a reliable transport protocol (RTP).

[0039] ULP can be implemented on any number of hardware or software devices, such as remote direct memory access (RDMA) or non-volatile memory standards (Non-Volatile Memory Express, NVMe), and can be responsible for implementing hardware / software interfaces, message processing, completion notification, end-to-end flow control, or any combination of the above. RTP can be used to track the reception of data packets at the target node. For example, for data packets pushed to the target node using RTP, the target node can send an acknowledgment message (ACK) to confirm the receipt and placement of the data packet. Receipt can be transmitted through the network connection 130, such as by using ULP or RTP. In other instances, the target node can send a negative acknowledgment (NACK) to indicate that the target node is not ready to receive the data packet, and the transmitting node should try to send the data packet again at a later time. In the case of receiving a NACK, the transmitting node can attempt to retransmit the data packet at a later time. A peer connection can be formed between corresponding protocols at each level through the network 100. In addition, within each node, the ULP of each node can be configured to communicate with the RTP of other nodes.

[0040] Among other factors, ACK and NACK signals may indicate whether there is congestion at the remote node, such as whether too many packets are transmitted from one or more transmitting nodes to the remote node in too short a period of time.To resolve congestion, the transmitting node may adjust the rate at which data packets are transmitted to the target node.

[0041] Figure 2 It is used in applications such as Figure 1 1 is a block diagram of a system 200 for performing congestion control within a communication network 100 of the present invention. Figure 2 The system 200 may be included in a first node (such as Figure 1 110) and may be configured to communicate with a second node in the communication network (such as Figure 1 The Node B 120 of the second node collects information and updates the rate of packet transmission to the second node based on the collected information to control congestion at the second node.

[0042] The primary data path 201 is configured to bidirectionally transmit data between a first node and a second node. At the first node, an incoming packet stream 202 may be received along the primary data path 201 and temporarily stored at a memory of the first node, which is located at Figure 2 204. The outgoing packet stream 204 may be transmitted along the primary data path 201 by a communication device of the first node, which is represented as a data buffer 210. Figure 2 denoted as transmitter 220.

[0043] Data packets received from the second node by the incoming packet stream 202 and stored in the data cache 210 can be analyzed to determine whether the rate of packet transmission to the second node should be updated. The analysis can be divided into two different stages. In the first stage, it is determined whether to initiate a rate update event. The first stage can be implemented using one or more computing devices 230 included in the hardware of the first node and communicatively coupled to the main data path 201. In the second stage, updated parameters for data packet transmission are determined in response to the initiation of the rate update event. The second stage can be implemented using a rate update engine 240 separated from the main data path 201. For example, data from the incoming packet stream 202 can be stored independently of the data cache 210, and the rate update engine 240 can analyze the separately stored data to determine the updated parameters for data packet transmission. Separating the congestion control process between these two stages helps to offload processing from the main data path and thus avoid congestion on the main data path. Other advantages of offloading the congestion control process include increased flexibility in defining congestion control algorithms and, in at least some examples, increased flexibility in controlling whether to initiate a rate update event. These and other aspects of the two-stage congestion control process are described in more detail herein.

[0044] Figure 3 yes Figure 2 A block diagram of a first example implementation of a system 200 is shown. Figure 3 In an example implementation, Figure 2 The buffer 210 and transmitter 220 are simplified into a single block called "transmit / buffer / receive" 310. Block 310 may include Figure 2 2 and 3. The block 310 may be a connection context cache capable of managing and tracking congestion indicators for use in various congestion control algorithms.

[0045] exist Figure 3 , the hardware of the first node is represented by a dashed box 320. Figure 3As can be seen, each of the first and second stages of the congestion control system is included in the hardware of the first node. For example, the rate update engine 340 can be a hardware engine executed using the processor of the first node and having instructions programmed at the memory of the first node. In this example, the data received through the main data path 301 can be stored in the cache of the box 310, which can be a connection context cache. One or more computing devices 330 of the first stage can analyze the stored data and can trigger the rate update engine 340 in response to the analysis. The rate update engine 340 can then initiate a rate update event in order to perform congestion control analysis separately from the processing performed at the main data path 301. The results of the congestion control analysis can then be fed from the rate update engine 340 to the box 310 to implement the updated parameters for data packet transmission.

[0046] Figure 4 is a block diagram of an example computing device included in the hardware of the first node, which may be configured to implement Figure 3 The congestion control system 300. Figure 4 In the example of , the computing device 330 of the first stage may include one or more processors 410, a memory 420, and an input / output component 450 for receiving and transmitting data with other components included in the hardware of the first node, such as the rate update engine 340, and components external to the first node. Figure 4 In the example of , the rate update engine of the second stage may include one or more processors 460, memory 470, and input / output components 490 for receiving and transmitting data with other components included in the hardware of the first node, such as the computing device 330 of the first stage, and components external to the first node. Although the processors and memories of blocks 330 and 340 are shown as being completely separate from each other, in some examples, the two blocks may share some or all of the processors, some or all of the memories, or any combination thereof.

[0047] The first stage memory 420 may include instructions 424, and may further include data 422 that the processor 410 can obtain, store, or modify in accordance with the instructions 424. For example, although the systems and methods are not limited to a particular data structure, the data 422 may be stored in a computer register, stored in a data store as a structure having a plurality of different fields and records, or stored in a file or buffer. The data 422 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, the data 422 may include information sufficient to identify the relevant information, such as a number, descriptive text, attribute codes, pointers, references to data stored in other memories, including other network locations, or information used by a function to calculate the relevant data.

[0048] The instructions 424 may be a set of instructions directly executed by the processor 410, such as machine code, or a set of instructions indirectly executed by the processor 410, such as a script. In this regard, the terms "instructions," "steps," and "programs" may be used interchangeably herein. The instructions 424 may be stored in an object code format for direct processing by the processor 410, or in other types of computer languages, including scripts or a collection of independent source code modules that are interpreted as required or compiled in advance.

[0049] The second stage memory 470 may also include data 472 and instructions 474. The data 472 and instructions 474 may operate according to the examples described in conjunction with the data and instructions of the first stage memory 420.

[0050] The processor 410 of the first stage 330 may be able to access the packet transmission data 432 from the data 422. The packet transmission data 432 may indicate a response of the second node to the data packet transmitted from the first node to the second node via the RTP layer. For example, the packet transmission data 432 may indicate whether an ACK packet or a NACK packet is received from the second node.

[0051] The processor 410 of the first stage 330 may also be able to access congestion control settings 434 from the data 422. The congestion control settings 434 may be one or more parameters for controlling data transmission between the first node and the remote second node in order to avoid or alleviate congestion at the second node. An example parameter may be an inter-packet gap for limiting the number of packets transmitted between the first and second nodes per round trip time (RTT). For example, when the second node is determined to be experiencing in-cast congestion, as indicated by the total number of unresolved packets transmitted to the second node via the RTP layer, the inter-packet gap may be reduced to less than 1 packet per RTT. Another example parameter may be an explicit congestion notification (ECN) received from the second node. The ECN may indicate that there is congestion at the second node without discarding packets transmitted by the first node. Another example parameter may be a congestion window size, which may be used to control the total number of unresolved packets per connection measured in RTT. Another example parameter may be a retransmission timeout (RTO), which is the amount of time a first node waits before retransmitting a packet to a second node via the RTP layer if an ACK is not received from the second node in response to the initial transmission of the packet.

[0052] Instructions 424 may include various algorithms for instructing the processor 410 of the first stage to perform operations according to the routines described herein. For example, instructions 424 may include a packet transmission routine 442 for sending packets from a first node to a second node via an RTP layer. For another example, instructions 424 may include a congestion indicator determination routine 444 for evaluating congestion at the second node based on data received via a primary data path, such as packet transmission data 432, and possibly in conjunction with a current congestion control setting 434. For another example, instructions 424 may include a rate update request routine 446 for triggering a rate update engine to initiate a rate update event. For another example, instructions 424 may include a request limiter routine 448 for determining whether a queue has received data for a rate update request based on the remaining available space in a data buffer that sends a rate update request to the rate update engine.

[0053] The processor 460 of the second stage 340 may be able to access a congestion indicator 482 from the data 472. The congestion indicator 482 may be provided by the processing device of the first stage so that the rate update engine of the second stage determines the updated parameters for the packet transmission to the second node. An example indicator may be a congestion window, which may indicate the total number of unresolved data packets that the first node transmits to the second node for which an ACK or NACK packet has not yet been received. Another example indicator may be a round trip delay, which is the time between the transmission of the data packet through the RTP layer and the receipt of the ACK packet in response. Other example indicators may include the total number of packets confirmed by the second node, the number of retransmissions, the number of clear congestion notifications received from the second node, or any combination thereof.

[0054] The processor 460 of the second stage 340 may also be able to store and access congestion control parameters in the data 472. The congestion control parameters may be parameters used by the computing device of the first stage 330 to set congestion control settings.

[0055] Figure 5 yes Figure 2 A block diagram of a second example implementation of a system 200 is shown. Figure 5 In an example implementation, Figure 2 The buffer 210 and transmitter 220 are simplified into a single block called "transmit / buffer / receive" 510. Block 510 may include Figure 2 The blocks of the buffer 210 and transmitter 220 in FIG. 5 have equivalent functions and can transmit and receive incoming data packet streams 502 and outgoing data packets 504 through the main data path 501 .

[0056] exist Figure 5 , the hardware of the first node is represented by a dashed box 520. Figure 5 As can be seen, the first stage of the congestion control system is included in the hardware of the first node, but at least some of the second stage of the congestion control system is managed by components separate from the hardware of the first node. In particular, rate update requests are managed by a software module, and dedicated hardware or hardware of the first node is not required to process rate update requests.

[0057] exist Figure 5 In the example of , one or more computing devices 530 of the first stage are connected to a mailbox mechanism 540 to forward rate update requests to a separate software module for processing. The mailbox mechanism includes a memory for temporarily storing rate update requests and associated information, such as a congestion indicator determined by the computing device 530 of the first stage. Figure 5In the example of , the mailbox mechanism 540 includes a first queue 542 or buffer for temporary storage. The mailbox mechanism 540 may also include a second queue 544 or buffer for temporarily storing the result of the rate update event, so that the result can be fed back to the hardware 520 of the first node and used to control the transmission of data packets along the data path 501. Figure 5 In the example of , the mailbox mechanism itself is included in the hardware 510 of the first node, but in other examples, memory separate from the hardware of the first node may be used to buffer the incoming stream of rate update results.

[0058] exist Figure 5 In the example of , the on-chip network 550 is used to interface the hardware portion 520 of the system 500 to the software portion. The on-chip network 550 can be connected to a memory 560 that is separate from the hardware 520 of the first node and can be accessed by the rate update engine software 570. The rate update engine software 570 can further have access to one or more processors 580 that are separate from the hardware 520 of the first node, whereby the instructions of the rate update engine software module can control the processor 580 to process the data stored in the memory 560 according to the programmed rate update algorithm, thereby deriving a rate update result from the input data. The interface provided by the on-chip network 550 can be bidirectional so as to provide the rate update result back to the hardware 520 of the first node.

[0059] Figure 6 6 is a diagram illustrating an example subsystem 600 for interfacing between a rate update scheduler 610 of hardware of a first node and a software-based rate update engine 620 stored separately from the hardware of the first node. A communication interface such as Figure 5 The memory of the memory 560 is used to temporarily store the rate update request sent from the first node to the software and the rate update result fed back from the software to the first node. Figure 6 In the example of , the memory is shown in the form of a first circular buffer 630 for storing rate update requests and a second circular buffer 640 for storing rate update results. Each buffer can be configured to act as a queue, whereby information is sequentially exchanged from the connection context cache of the hardware of the first node to the rate update engine software, and vice versa. The first buffer is referred to herein as the event queue, and the second buffer is referred to as the result queue.

[0060] Each buffer 630, 640 can be stored in a memory space associated with the software module 570. The event queue can include a tail pointer 632 for adding rate update requests to the queue and a head pointer 634 for moving queued requests forward to the software module. Similarly, the result queue can include a tail pointer 642 for adding rate update results to the queue and a head pointer 644 for moving queued results forward to the first node. Each of the pointers 632, 634, 642, 644 and the overall size of each buffer can be stored in a control / status register of the memory of the first node, so that the hardware read / write interface 615 can push requests to the correct unit of the corresponding buffer and pull results from there. The head pointer 634 of the event queue can also be stored in the memory of the read / write interface 625 of the rate update engine 620, which can pull requests from the correct unit of the event queue 640. As the read / write interface 625 of the rate update engine 620 pulls requests from or pushes results to the corresponding buffer, an interface such as the Advanced eXtensible Interface (AXI) may be used to update the pointers stored in the CSR of the rate update scheduler 610. In operation, the head pointer and tail pointer may be advanced by the rate update scheduler 610 and the rate update engine 620 as data items are added and removed from the queue, and the AXI write interface may be used to maintain consistency between the records in the memory of the rate update engine 620 and the rate update scheduler 610.

[0061] return Figure 5 , the software rate update engine can be implemented on an ARM compute complex (ACC) and can be designed to handle up to 15 million congestion control events per second. In contrast, an implementation of the rate update engine as an additional hardware block of the first node can be designed to support up to 200 million congestion control events. Therefore, when congestion control is implemented in software, it is necessary to be more selective about what events trigger rate update events and what events do not trigger rate update events. In addition, because the communication between the hardware of the first node and the software of the rate update engine is maintained by a data buffer with limited memory space, the degree of selectivity about what events trigger rate update events and what events do not trigger rate update events can vary depending on how full the buffer is at a given time.

[0062] By storing buffer pointer data and size data in a control / status register of the first node, the first node may track even how full the queue is at a given time and provide dynamic control over when future events should trigger another rate update event. Figure 7An example routine for limiting the requests sent from the rate update scheduler to the rate update engine is illustrated, which was previously described in Figure 4 is referred to as request limiter routine 448.

[0063] Certain predicates or conditions may be selected to determine whether a rate update is required, or more precisely, the degree of urgency of a rate update. A first example predicate may be based on the amount of time since a previous rate update request, since the urgency of a new rate update increases as the time since the previous rate update request increases. A second example predicate may be based on the amount of time since the second node received a previous ACK packet, since the urgency of a new rate update increases as the time since the previous ACK packet was received increases. These conditions are referred to herein as predicate 1 and predicate 2, or simply as P1 and P2, respectively.

[0064] exist Figure 7 In the example of , the event queue is illustrated by a horizontal bar, for which the horizontal axis of the bar starts from empty at the right end to full at the left end to indicate how full the queue is. The bar is divided into multiple segments, each segment being associated with a different set of conditions for pushing rate update events onto the event queue. The first segment is shown as being between empty and threshold 1. When the fullness of the queue is between these two numbers, all request pairs are pushed to the queue. The second segment is shown as being between threshold 1 and threshold 2. When the fullness of the queue is between these two numbers, requests can be pushed to the queue when either P1 or P2 is true, indicating the urgency of the rate update due to the time since the previous update or the time since the previous ACK packet. The third segment is shown as being between threshold 2 and threshold 3. When the fullness of the queue is between these two numbers, requests can be pushed to the queue when P1 is true, indicating the urgency of the rate update due only to the time since the previous update, without considering how long it has been since the previous ACK packet was received. The fourth segment is shown as being between threshold 3 and full. When the fullness of the queue is between these two numbers, fewer requests can be pushed to the queue to save space in the queue, and the requests that are pushed are limited to the time when both P1 and P2 are true.

[0065] In some examples, the request limiter routine can be run on the software side, and all requests can be pushed to the software and subsequently discarded based on the varying urgency levels as described herein. However, since the software may have limited processing resources, it is often advantageous to additionally operate the request limiter routine from the hardware side. This frees up the rate update engine software module to handle more important events and avoid exceeding the limits of the software module's processing capabilities. Example Method

[0066] The routines performed by the first and second stages of the congestion control system and other routines are combined Figure 8 and 9 It should be understood that the routines described herein are only examples, and in other examples, some steps may be added, deleted, replaced or reordered.

[0067] Figure 8 8 is a flow chart illustrating an example routine 800 for congestion control. Certain steps of the routine are described as being performed by one or more processors of a computing device in a first phase. Other steps are described as being performed by one or more processors of a computing device in a second phase. However, in other example routines, the processor responsible for at least some of these steps may be switched from the first phase to the second phase, or vice versa.

[0068] At block 810, one or more processors of the first stage may analyze data packets received from the remote node via the communication protocol layer. At block 820, one or more processors of the first stage may determine a congestion indicator based on the analysis of block 810. The congestion indicator may include, but is not limited to, a congestion window, a round trip delay, a total number of packets acknowledged by the remote node, a number of retransmissions, a number of explicit congestion notifications received from the remote node, or any combination thereof.

[0069] At block 830, one or more processors of the first stage may determine whether to send a rate update request to the second stage. If it is determined not to send a rate update request, operations end at block 840, where no rate update request is sent and no rate update event is performed.

[0070] Alternatively, if it is determined to send a rate update request, the operation can continue at block 850, whereby the rate update request and the congestion indicator can be transmitted from the processor of the first stage to one or more processors of the rate update engine of the second stage. In some embodiments, sending the rate update request can involve instructing another hardware module of the first node to process the packet transmission event and the congestion indicator. In other embodiments, sending the rate update request can involve writing the packet transmission event and the congestion indicator to a mailbox queue. This can be implemented using a coherent mesh network such as CMN-600 to maintain cache coherence throughout the system.

[0071] At block 850, the one or more processors of the second stage may determine congestion control parameters based on the congestion indicator. The congestion control parameters may include, but are not limited to, inter-packet gaps, retransmission timeouts, or any combination thereof.

[0072] At block 860, the one or more processors of the second stage may output the congestion control results back to the processors of the first stage for implementation. The congestion control results may include the congestion control parameters determined at block 850, so that congestion control implementation is performed in accordance with the determined congestion control parameters.

[0073] In box 870, one or more processors of the first stage can receive congestion control results from the rate update engine. In box 880, the congestion control results can be implemented, whereby one or more processors of the first stage that communicate with the remote node using the data path control the transmission of data packets to the remote node through the data path. In the case where the inter-packet gap increases or the retransmission timeout increases, the control can cause the overall frequency of transmitting packets to decrease. This in turn can alleviate congestion at the remote node, such as in-cast congestion. Conversely, in the case where the inter-packet gap decreases or the retransmission timeout decreases, the control can cause the overall connection traffic on the data path to decrease. This may be the result that congestion such as in-cast congestion at the remote node has been alleviated and the connection traffic on the data path is allowed to increase.

[0074] Fig. 9 It is a graphic Figure 8 830 of the example subroutine 900, which determines whether to send a rate update request. The operation of the routine 900 can start at block 820, and then determine whether to send a rate update request. Figure 8 The operations of block 840 or 850 continue.

[0075] At block 910, one or more processors of the first stage may determine whether a packet transmission event has occurred. In some examples, a packet transmission event may include receiving an ACK packet or a NACK packet from a remote node. Also, in some examples, a packet transmission event may include a retransmission of a packet to a remote node, such as in response to receiving a NACK packet from a remote node that relates to a retransmitted packet. If it is determined that a packet transmission event has not occurred, operations may end at block 840. Alternatively, if it is determined that a packet transmission event has occurred, operations may optionally continue at block 850 by transmitting a rate update request to a second stage processor.

[0076] In some examples, subroutine 900 may include additional steps for limiting the frequency of rate update requests, particularly when requests are provided to a rate update engine software module having relatively limited capacity compared to a hardware implementation at a transmitting node for performing rate updates.

[0077] A first condition or predicate may be determined by one or more processors of the first stage at block 920. The first condition may indicate whether the time since a previous rate update was performed exceeds a first predetermined threshold, also referred to as a latency threshold.

[0078] At block 930, a second condition or predicate may be determined by one or more processors of the first stage. The second condition may indicate whether the time since the remote node received a previous ACK packet exceeds a second predetermined threshold, also referred to as a throughput threshold.

[0079] At block 940, the one or more processors of the first stage may determine the amount of remaining space in the event queue or buffer for sending rate update requests to the rate update engine. The remaining space may be determined based on information stored in a control / status register of the transmitting node. For example, in the case of a ring buffer, the remaining space may be determined based on the difference between the buffer size and the distance between the head pointer and the tail pointer of the ring buffer.

[0080] Based on the determination of box 940, the operation can proceed to any one of several boxes, each box being associated with a different set of judgment formulas or conditions by which it can be determined whether to send a rate update request to the rate update engine. The condition set may include but is not limited to: no condition, only the first condition, only the second condition, any one of the first or second conditions, or both the first condition and the second condition. In addition, different first and second thresholds can be specified for different condition sets. For example, the first delay threshold can be used for rate update event limitation when the buffer is relatively empty, but the longer second delay threshold can be used for rate update event limitation when the buffer is relatively full. For another example, the first throughput threshold can be used for rate update event limitation when the buffer is relatively empty, but the longer second throughput threshold can be used for rate update event limitation when the buffer is relatively full.

[0081] exist Fig. 9 In the specific example of FIG. 9A , the operation of subroutine 900 is divided among four condition sets.

[0082] First Condition Set: If the capacity C of the buffer is less than the first threshold T1, then operations may continue at block 950, whereby the one or more processors of the first stage determine to continue operations at block 850 and send a rate update request to the buffer.

[0083] Second set of conditions: If the capacity C of the buffer is greater than the first threshold T1 but less than the second threshold T2, the operation can continue at block 960, whereby the one or more processors of the first stage determine whether the first or second condition (P1 or P2) has been met. If P1 or P2 has been met, the operation can continue at block 950 and subsequently 850 as described above. Otherwise, if both P1 and P2 have not been met, the operation can end at block 840.

[0084] Third condition set: If the capacity C of the buffer is greater than the second threshold T2 but less than the third threshold T3, the operation can continue at block 970, and the one or more processors of the first stage determine whether the first condition P1 has been met. If P1 has been met, the operation can continue at block 950 and subsequently 850 as described above. Otherwise, if P1 has not been met, the operation can end at block 840.

[0085] Fourth condition set: If the capacity C of the buffer is greater than the second threshold value T3, the operation can continue at block 960, whereby the one or more processors of the first stage determine whether both the first and second conditions (P1 and P2) have been met. If both P1 and P2 have been met, the operation can continue at block 950 and subsequently 850 as described above. Otherwise, if both P1 and P2 have not been met, the operation can end at block 840.

[0086] For each instance where the operation ends at block 840, ending the operation may result in fewer rate update requests being sent to the offline engine. This may allow the system to absorb bursts and temporary reverse pressures between the primary data path and the offline engine, thereby ensuring that the primary data path does not stall.

[0087] Figure 8 and 9 The routine may be repeatedly initiated by one or more processors of the computing device of the first stage. For example, the routine may be initiated each time a data packet is received from a remote node through the data path.

[0088] The routines described above may be advantageous for offloading congestion control algorithms in general, and more specifically, for offloading the determination of processing rate update results from the primary data path of a communication protocol, leaving other processing modules of the hardware of the communication node or a connected software module to perform the processing. Furthermore, because the congestion control algorithm is separated from the hardware of the primary data path, the designated congestion control engine may be more easily adapted or designed to execute any number of congestion control protocols, including but not limited to the GCN, Swift, or BBR protocols.

[0089] Although the technology has been described herein with reference to specific embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it is to be understood that various modifications may be made to the illustrative embodiments and that other deployment forms may be conceived without departing from the spirit and scope of the technology as defined in the appended claims.

[0090] Most of the above replaceable examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since other changes and combinations of the features discussed above can be adopted without departing from the subject matter defined by the claims, the above description of the embodiment should be considered as an illustration of the subject matter defined by the claims rather than a limitation. As an example, the aforementioned operations do not necessarily have to be performed in the exact order described above. On the contrary, the various steps can be processed in different orders, such as reverse processing or simultaneous processing. Unless otherwise indicated, steps can also be omitted. In addition, the examples described herein and the clauses worded as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, the examples are intended to be only one of many possible embodiments. In addition, the same figure marks in different figures can identify the same or similar elements.

Claims

1. A system comprising: a buffer located between a packet data path of a communication network and a packet transmission rate controller, the packet transmission rate controller being configured to control a transmission rate of data packets on the packet data path based on one or more congestion indicators; as well as One or more processors configured to: identifying a rate update event based on analysis of the data packets, wherein the rate update event is associated with the one or more congestion indicators; in response to identifying the rate update event, determining whether to transmit a rate update request to the packet transmission rate controller based at least in part on a remaining available capacity of the buffer, the rate update request including the one or more congestion indicators; In response to the remaining available capacity of the buffer being less than a threshold, refraining from transmitting the rate update request; and In response to the remaining available capacity of the buffer being greater than or equal to a threshold, the rate update request is transmitted.

2. The system according to claim 1, wherein The buffer is located between the packet data path and the on-chip network, wherein The network on chip is in communication with the packet transmission rate controller, and wherein the rate update request is temporarily stored at the buffer before being sent to the network on chip.

3. The system according to claim 1, wherein: The one or more processors are configured to receive a congestion control result from the packet transmission rate controller indicating an updated packet transmission rate, the updated packet transmission rate being based on one or more congestion control parameters.

4. The system according to claim 1, wherein: The one or more congestion indicators indicate network congestion for data packets transmitted via the reliable transport protocol layer.

5. The system according to claim 1, wherein: The rate update event is one of an acknowledgement message, a non-acknowledgement message, or a packet retransmission.

6. The system according to claim 1, wherein: The one or more congestion indicators include at least one of the following for data packets transmitted to the second node: a congestion window, a retransmission timeout, a round trip delay, a number of packets acknowledged by the second node, a number of retransmissions, or a number of explicit congestion notifications received.

7. The system according to claim 1, wherein: The buffer comprises: a first queue configured to queue input data provided to the packet transmission rate controller; and A second queue is configured to queue output data from the packet transmission rate controller.

8. The system according to claim 7, wherein: The first queue is a first circular buffer having a first head pointer and a first tail pointer, and wherein the second queue is a second circular buffer having a second head pointer and a second tail pointer.

9. The system according to claim 8, wherein: The one or more processors are configured to push a rate update request to the first ring buffer according to the first tail pointer, and read a congestion control result generated by the packet transmission rate controller from the second ring buffer according to the second head pointer.

10. The system according to claim 1, wherein: The one or more processors are configured to determine, in response to identifying the rate update event, whether to transmit a rate update request to the packet transmission rate controller based on comparing the remaining available capacity of the buffer to each of a first threshold capacity value and a second threshold capacity value that is less than the first threshold capacity value.

11. The system according to claim 10, wherein: The one or more processors are configured to: transmitting the rate update request in response to the remaining available capacity of the buffer being greater than or equal to the first threshold capacity value; in response to the remaining available capacity of the buffer being less than the first threshold capacity value but greater than the second threshold capacity value, determining whether to transmit the rate update request based on a first set of criteria; as well as In response to the remaining available capacity of the buffer being less than the second threshold capacity value, determining whether to transmit the rate update request is based on a second set of criteria.

12. The system according to claim 11, wherein: The first set of criteria includes at least one of the following: The amount of time since the previous rate update exceeds a predetermined latency threshold; or The number of acknowledgement (ACK) packets received for transmitted data packets exceeds a predetermined throughput threshold.

13. The system according to claim 12, wherein: The second set of criteria includes the amount of time since the previous rate update exceeding the predetermined latency threshold.

14. The system according to claim 12, wherein: The second set of criteria includes the following two items: said amount of time since said previous rate update exceeds said predetermined latency threshold; or The number of acknowledgement (ACK) packets received for transmitted data packets exceeds the predetermined throughput threshold.

15. The system of claim 1, wherein: The one or more processors are configured to determine, in response to identifying the rate update event, whether to transmit a rate update request to the packet transmission rate controller based on comparing the remaining available capacity of the buffer with each of a first threshold capacity value, a second threshold capacity value less than the first threshold capacity value, and a third threshold capacity value less than the second threshold capacity value.

16. The system of claim 15, wherein: The one or more processors are configured to: transmitting the rate update request in response to the remaining available capacity of the buffer being greater than or equal to the first threshold capacity value; in response to the remaining available capacity of the buffer being less than the first threshold capacity value but greater than the second threshold capacity value, determining whether to transmit the rate update request based on a first set of criteria; in response to the remaining available capacity of the buffer being less than the second threshold capacity value but greater than the third threshold capacity value, determining whether to transmit the rate update request based on a second set of criteria; as well as In response to the remaining available capacity of the buffer being less than the third threshold capacity value, determining whether to transmit the rate update request is based on a third set of criteria.

17. The system of claim 16, wherein: The first set of criteria includes at least one of: an amount of time since a previous rate update exceeds a predetermined latency threshold or a number of acknowledgement (ACK) packets received for transmitted data packets exceeds a predetermined throughput threshold; The second set of criteria includes the amount of time since the previous rate update exceeding the predetermined latency threshold; and The third set of criteria includes both the amount of time since the previous rate update exceeding the predetermined latency threshold and the number of acknowledgement (ACK) packets received for transmitted data packets exceeding the predetermined throughput threshold.

18. A method comprising: identifying, by one or more processors, a rate update event based on analysis of data packets transmitted on a packet data path of a communication network, wherein the rate update event is associated with one or more congestion indicators; and determining, by the one or more processors, in response to identifying the rate update event, based on the one or more congestion indicators, whether to transmit a rate update request to a packet transmission rate controller to control a transmission rate of data packets on the packet data path, wherein the determination is based at least in part on a remaining available capacity of a buffer between the packet data path and the packet transmission rate controller; In response to the remaining available capacity of the buffer being less than a threshold, refraining from transmitting the rate update request; and In response to the remaining available capacity of the buffer being greater than or equal to a threshold, the rate update request is transmitted.

19. The method according to claim 18, wherein: Determining whether to transmit a rate update request to the packet transmission rate controller is further based on comparing the remaining available capacity of the buffer to each of a first threshold capacity value, a second threshold capacity value less than the first threshold capacity value, and a third threshold capacity value less than the second threshold capacity value.

20. The method according to claim 19, wherein: Determining whether to transmit a rate update request to the packet transmission rate controller includes: responsive to the remaining available capacity of the buffer being greater than or equal to the first threshold capacity value, transmitting a rate update request; in response to the remaining available capacity of the buffer being less than the first threshold capacity value but greater than the second threshold capacity value, determining whether to transmit the rate update request based on whether an amount of time since a previous rate update exceeds a predetermined latency threshold or a number of acknowledgment (ACK) packets received for transmitted data packets exceeds a predetermined throughput threshold; responsive to the remaining available capacity of the buffer being less than the second threshold capacity value but greater than the third threshold capacity value, determining whether to transmit the rate update request based on whether the amount of time since the previous rate update exceeds the predetermined latency threshold; and In response to the remaining available capacity of the buffer being less than the third threshold capacity value, determining whether to transmit the rate update request is based on both whether the amount of time since the previous rate update exceeds the predetermined latency threshold and whether the number of acknowledgment (ACK) packets received for transmitted data packets exceeds the predetermined throughput threshold.

Images