Reliable transport protocol and hardware architecture for data center networking
By introducing request-based push transactions and sliding window mechanisms in the data center, the problems of aggregation broadcast congestion and tail delay in the data center are solved, and data transmission efficiency and connection reliability are improved.
Patent Information
- Application Number
- CN202411308868.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2020-03-16
- Filing Date
- 2020-11-13
- Publication Date
- 2024-12-06
AI Technical Summary
The existing communication protocols used in data centers are prone to cause congestion and long-tail delay in many-to-one communication mode, affecting data transmission efficiency.
By introducing a request-based push transaction mechanism, the initiator entity allows the order and size of data transmission, and uses sliding windows and bitmaps to support out-of-order transactions, reducing retransmission attempts and preventing connection disconnection.
It effectively reduces congestion and tail delay, improves data transmission efficiency, and achieves more flexible and reliable connections.
Smart Images

Figure CN119094513A_ABST
Abstract
Description
Description of the case This application is a divisional application of the patent application with application number 202011269309.2 submitted to the China Patent Office on November 13, 2020, and the invention name is "Reliable transmission protocol and hardware architecture for data center networking". CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 947,036, filed on December 12, 2019, the disclosure of which is incorporated herein by reference. Technical Field
[0002] The present disclosure relates to a reliable transport protocol and hardware architecture for data center networking. Background Art
[0003] The Internet Protocol Suite is a set of communication protocols for servicing the transmission of data between two devices that communicate information over the Internet or other computer networks. The Transmission Control Protocol ("TCP") is part of the Internet Protocol Suite and provides connection-oriented, reliable, and ordered delivery of a stream of data packets between, for example, a web browser application running on a client device and a web server application running on a server device, over a local area network or a wide area network. Currently, data centers that use communication protocols such as TCP encounter certain problems. For example, incast is a common many-to-one communication mode in data centers, which can cause incast congestion when multiple synchronous computing devices send data to the same recipient computing device in parallel. In addition, because current communication protocols require packets to be delivered in order over a connection, long tail latency (which is the amount of time used for the last few packets in a series of packets to be transmitted) may prevent the next series of packets from being transmitted. Summary of the invention
[0004] The present disclosure provides: an initiator entity determines that outgoing data is to be transmitted to a target entity; the initiator entity transmits a solicited push request requesting that the outgoing data be placed at the target entity to the target entity; the initiator entity receives a push authorization in response to the solicited push request from the target entity; and the initiator entity transmits the outgoing data to be placed at the target entity to the target entity in response to the push authorization.
[0005] The method may further include: determining, by the initiator entity, that the size of the outgoing data satisfies a predetermined threshold, wherein transmitting the solicited push request is based on the determination that the size of the outgoing data satisfies the predetermined threshold. The push request may originate from an upper layer protocol of the initiator entity, and based on the upper layer protocol of the initiator entity, a reliable transport protocol layer of the initiator entity transmits the solicited push request as a packet through a connection between the initiator entity and the target entity.
[0006] The method may further include: determining, by the initiator entity, that the size of the outgoing data does not meet a predetermined threshold; and transmitting, by the initiator entity, the outgoing data to be placed at the target entity without sending the solicited push request or receiving a push authorization. The push request may originate from an upper layer protocol of the initiator entity, based on which a reliable transport protocol layer of the initiator entity sends the transmission data as packets over a connection between the initiator entity and the target entity.
[0007] The method may further include receiving, by the initiator entity, an acknowledgement from the target entity indicating that the outgoing data was received and placed at the target entity.
[0008] The method may further include: determining, by the initiator entity, that incoming data is needed from the target entity; transmitting, by the initiator entity, to the target entity a pull request requesting that the incoming data be transmitted to the initiator entity; receiving, by the initiator entity, from the target entity the incoming data in response to the pull request. The method may further include scheduling, by the initiator entity, the pull request for the incoming data based on one or more congestion parameters.
[0009] The present disclosure further provides: a sender entity transmits a plurality of packets in a first order through a connection to a receiver entity; the sender entity maintains at least one sliding window comprising a plurality of bits, wherein each bit of the sliding window represents a corresponding packet among the plurality of packets; the sender entity receives one or more confirmations indicating that the receiver entity has received one or more of the plurality of packets, each confirmation referencing a corresponding packet among the plurality of packets, wherein the confirmations are received in a second order different from the first order; and the sender entity modifies the value of one or more bits among the plurality of bits in the sliding window corresponding to the one or more received confirmations.
[0010] The method may further include adjusting, by the sender entity, a size of the sliding window based on one or more congestion parameters.
[0011] The plurality of packets may include one or more of: a request for a data packet, a data packet, an acknowledgment. The at least one sliding window may include a request sliding window. The at least one sliding window may include a data sliding window. The plurality of packets may include at least one data packet in response to a pull request. The plurality of packets may include at least one push authorization packet in response to a solicited push request.
[0012] The present disclosure further provides: an initiator entity transmits multiple packets to a target entity via a connection; the initiator entity determines that neither a confirmation nor a negative confirmation is received in response to a specific packet among the multiple packets within a predetermined time period; the initiator entity retransmits the specific packet to the target entity based on the determination; the initiator entity receives a negative confirmation from the target entity in response to the retransmission; and the initiator entity determines whether to wait for confirmation of the specific packet or resynchronize based on the negative confirmation.
[0013] The method may further include: determining, by the initiator entity, that the negative acknowledgement indicates that the target entity is not ready for the particular packet; and in response to the negative acknowledgement, waiting, by the initiator entity, for an acknowledgement from the target entity without retransmitting the particular packet to the target entity again.
[0014] The method may further include: determining, by the initiator entity, that the negative acknowledgement indicates that the target entity has erroneously completed an operation on the particular packet; and transmitting, by the initiator entity, a resynchronization packet to the target entity without disconnecting the connection. The method may further include: receiving, by the initiator entity, an acknowledgement in response to the resynchronization packet from the target entity; and transmitting, by the initiator entity, a subsequent plurality of packets in response to the acknowledgement of the resynchronization packet.
[0015] The plurality of packets may be transmitted according to a request from an upper layer protocol of the initiator entity, and retransmission of the specific packet is performed by a reliable transport protocol layer of the initiator entity. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a schematic diagram of a network in accordance with aspects of the present technique.
[0017] Figure 2 is a block diagram of an exemplary system in accordance with aspects of the present technology.
[0018] Figure 3 is a block diagram of communication layers in accordance with aspects of the present technology.
[0019] Figure 4 An exemplary sliding window in accordance with aspects of the present technique is shown.
[0020] Figure 5is an exemplary timing diagram of a pull transaction in accordance with aspects of the present technique.
[0021] Figure 6 is an exemplary timing diagram of an unsolicited push transaction in accordance with aspects of the present technique.
[0022] Figure 7 is an exemplary timing diagram of a solicited push transaction in accordance with aspects of the present technique.
[0023] Figure 8 is an exemplary timing diagram of ordered transactions in accordance with aspects of the present technique.
[0024] Fig. 9 is an exemplary timing diagram of out-of-order transactions in accordance with aspects of the present technique.
[0025] Fig.10 is an exemplary timing diagram involving negative acknowledgements in accordance with aspects of the present technique.
[0026] Fig.11 is an exemplary timing diagram involving error completion confirmation in accordance with aspects of the present technique.
[0027] Fig. 12A An exemplary basic header in accordance with aspects of the present technique is shown.
[0028] Fig. 12B A method for using a Fig.10 An exemplary definition of an exemplary basic header.
[0029] Fig.13A , 13B 13C show exemplary request and authorization headers according to aspects of the present technology.
[0030] Fig.13D , 13E and 13F show the use of Fig.13A , 13B and exemplary definitions of exemplary headers for 13C.
[0031] Fig.14A , 14B 14C show an exemplary data header according to aspects of the present technique.
[0032] Fig.15A An exemplary re-sync header in accordance with aspects of the present technique is shown.
[0033] Fig. 15B A method for using a Fig.15A An exemplary definition of an exemplary resynchronization header.
[0034] Fig.16A An exemplary negative acknowledgment header in accordance with aspects of the present technique is shown.
[0035] Fig. 16B A method for using a Fig.16A An exemplary definition of an exemplary negative acknowledgement header.
[0036] Fig.17A An exemplary error completion confirmation header in accordance with aspects of the present technique is shown.
[0037] Fig. 17B A method for using a Fig.17A An exemplary definition of an exemplary error completion confirmation header.
[0038] Fig.18 is an exemplary flow chart according to aspects of the present technology. DETAILED DESCRIPTION
[0039] The technology generally relates to a communication protocol for reliably transmitting packets over a connection. The technology provides request-based push transactions that provide recipient entity control over incoming data, thereby reducing incast congestion and tail latency. The technology further supports out-of-order transaction processing over the connection using a sliding window and a bitmap, which can improve the overall efficiency of processing packets over the connection. The technology further provides handling of failed transmissions, which reduces retransmission attempts and uses resynchronization to prevent connection disconnection, resulting in a more resilient connection.
[0040] Figure 1 An exemplary network 100 is shown. Network 100 includes various entities, such as entity A, entity B, and entity C. In order to communicate with each other, connections are formed between the entities, such as connection 110 between entities A and B, and connection 120 between entities A and C. The entities can communicate over the connection using one or more protocols. For example, a reliable transport (RT) protocol is a protocol that tells a sender whether data was successfully delivered to the intended recipient. The sender and receiver are considered peers of the communication protocol, so entities A and B can be reliable transport peers, and entities A and C can be reliable transport peers. A connection using a reliable transport protocol is an end-to-end construction that describes a bidirectional communication channel between two reliable transport peers.
[0041] A connection may be identified by a pair of connection IDs ("CIDs"), one for each direction of communication. The CIDs may be assigned by the recipient entity during the connection establishment process and have no global meaning outside of the parties involved. Thus, a connection 110 between entities A and B may have a CID value of 5 for the direction from A to B, and a CID value of 10 for the direction from B to A. A connection 120 between entities A and C may have a CID value of 5 for the direction from A to C, and a CID value of 11 for the direction from C to A. Furthermore, the CIDs assigned by the entities, or the "source CIDs" of the entities, must have different values. Thus, in the example shown, the CID assigned by entity A, or the source CID of entity A, has different values of 10 and 11. In contrast, the "destination CID" of an entity is assigned by other entities and may have the same value. Thus, in the example shown, the destination CID of entity A is assigned by entities B and C, respectively, which may have the same value of 5.
[0042] Packets may be transmitted over a connection between entities. In this regard, a packet is the basic unit of communication across a connection. A packet may have a predetermined size, such as a length up to a maximum transmission unit ("MTU"). A packet may have a header that includes information about the packet and its transmission, as well as a payload of data. To ensure reliable transmission, a reliable transmission packet may include a destination CID, such as in a header. For example, when entity B receives a packet over connection 110 with a destination CID of 5, entity B may identify the packet as coming from entity A, and may then inform A that the packet has been received by sending an acknowledgment referencing the packet and its connection 110 with a CID of 5. The acknowledgment itself may be sent as a packet including a destination CID of 10.
[0043] Entities A, B, and C may be any type of device capable of communicating over a network, such as a personal computing device, a server computing device, a mobile device, a wearable device, a virtual machine, etc. Figure 2 2 is a block diagram of some components of an exemplary system 200 that can communicate using a reliable transport protocol. System 200 includes at least two entities with one or more connections between them. This should not be considered to limit the scope of the present disclosure or the usefulness of the features described herein. In this example, system 200 is shown as having two entities, one or more computing devices 210 and one or more computing devices 260, with connection 250 between them. For example, computing device 210 can be Figure 1 Entity A, and computing device 260 may be Figure 1 Entity B, and connection 250 may be Figure 1The computing devices 210 and 260 may be configured with similar components as shown, or may include other and / or different components. In the example shown, the computing device 210 includes one or more processors 220 and a memory 230.
[0044] The one or more processors 220 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a dedicated component, such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, the one or more computing devices 210 may include dedicated hardware components that perform specific computing processes.
[0045] The memory 230 may be any non-transitory type capable of storing information accessible by a processor, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writeable and read-only memory. The memory 230 of the computing device 210 may store information accessible by one or more processors 220, including data 232 and instructions 234.
[0046] The memory 230 may include data 232 that may be retrieved, manipulated, or stored by the processor 220. For example, as described in reference Figure 1 and Figure 3-17B The data such as communication protocols, connection information such as CID, header definitions, etc. can be retrieved, manipulated or stored by the processor 220.
[0047] The memory 230 of the computing device 210 may also store instructions 234 that can be executed by the one or more processors 220. For example, based on the instructions 234 and data 232 in the memory 230, one or more processors 220 may perform operations such as referring to Figure 1 and 3 -11 The instructions of the communication protocol and Fig.18 Flowchart of the process.
[0048] Data 232 may be retrieved, stored, or modified by one or more processors 220 in accordance with instructions 234. For example, although the subject matter described herein is not limited to any particular data structure, the data may be stored in computer registers, in a relational database as a table with many different fields and records, or in an XML document. The data may also be formatted in any computing device readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, the data may include any information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other storage (such as at other network locations), or information that can be used by a function that calculates the relevant data.
[0049] Instructions 234 may be any set of instructions to be executed directly by one or more processors, such as machine code, or any set of instructions to be executed indirectly, such as a script. In this regard, the terms "instructions," "applications," "steps," and "programs" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including a collection of scripts or independent source code modules that are interpreted on demand or compiled in advance.
[0050] Although not shown, the computing device 210 may further include other components that are typically present in a general-purpose computing device. For example, the computing device 210 may include output devices such as a display (e.g., a monitor with a screen, a touch screen, a projector, a television, or other device operable to display information), a speaker, tactile feedback, etc. The computing device 210 may also include user input devices such as a mouse, a keyboard, a touch screen, a microphone, a sensor, etc.
[0051] although Figure 2 The processor, memory, and other elements of the computing device 210 are functionally shown as being located in the same block, but the processor, computer computing device, or memory may actually include multiple processors, computers, computing devices, or memories, which may or may not be stored in the same physical housing. For example, the memory may be a hard drive or other storage medium located in a housing different from the computing device 210. Therefore, references to processors, computers, computing devices, or memories will be understood to include references to a collection of processors, computers, computing devices, or memories that may or may not run in parallel. For example, the computing device 210 may include a server computing device that operates as a load balancing server farm, a distributed system, and the like. In addition, although some of the functions described below are indicated as occurring on a single computing device with a single processor, various aspects of the subject matter described herein may be implemented by multiple computing devices, for example, by transmitting information over a network.
[0052] Computing device 210 may be able to communicate directly and indirectly with other entities of the network, such as computing device 260. Computing devices 210 and 260 may be interconnected using various protocols and systems, such that computing devices in the network may be part of the Internet, the World Wide Web, a specific intranet, a wide area network, or a local area network. Computing devices in the network may utilize standard communication protocols, such as Ethernet, WiFi, and HTTP, one or more company-proprietary protocols, and various combinations of the foregoing. Although certain advantages are obtained when transmitting or receiving information as described above, other aspects of the subject matter described herein are not limited to any particular manner of information transmission.
[0053] return Figure 1, packets may be transmitted between entities A, B, and / or C over the connection using one or more communication protocols. Figure 3 An exemplary communication protocol system 300 is shown. The communication protocol system 300 may be implemented, for example, by Figure 2 processors 220 and 270, two or more entities in a network, such as Figure 1 The communication protocol may be implemented on two or more entities A, B, and C of the network 100. As shown, each entity may include multiple layers of communication protocols. For example, entity A may include an upper layer protocol ("ULP") 310 and a reliable transport ("RT") protocol 330, while entity B may include an upper layer protocol 320 and a reliable transport protocol layer 340. Peers may be formed between the protocols of each layer. Therefore, ULP 310 and ULP 320 are ULP peers, and reliable transport protocol layer 330 and reliable transport protocol layer 340 are RT peers. Further as shown, within each entity, the upper layer protocol is configured to communicate with the reliable transport protocol.
[0054] As reference Figure 4-11 As described, the upper layer protocols 310, 320 may be responsible for implementing hardware / software interfaces, message processing, completion notifications, and / or end-to-end flow control. The upper layer protocol may be implemented on any of a plurality of hardware or software devices. For example, the upper layer protocol may be implemented as remote direct memory access ("RDMA"). As another example, the upper layer protocol may be implemented as a non-volatile memory interface specification ("NVMe").
[0055] Also refer to Figure 4-11 As shown, the reliable transport protocol 330, 340 can be responsible for reliable delivery of packets, congestion control, admission control and / or ordered or unordered delivery of packets. Each reliable transport protocol 330, 340 can be logically divided into two sublayers of the protocol. Therefore, as shown, the reliable transport protocol layer 330 is divided into a request sublayer 332 responsible for endpoint admission control and, optionally, ordered delivery of packets, and a sliding window sublayer 334 responsible for end-to-end reliable delivery and congestion control. Similarly, the reliable transport protocol layer 340 is also divided into a request sublayer 342 and a sliding window sublayer 344.
[0056] Figure 4 Exemplary sliding windows 410 and 420 are shown. Sliding windows 410 and 420 are used by entities to track a predetermined number of packets to be transmitted and acknowledged over a connection. For example, entity A may use TX sliding window 410 for tracking packets sent to entity B over connection 110, and use RX sliding window 420 for tracking packets received from entity B. Thus, sliding windows 410 and 420 may be used as Figure 3The TX sliding window and the RX sliding window may have different sizes as shown, or may alternatively have the same size.
[0057] Referring to the TX sliding window 410, in order to track packets, sender entity A assigns a packet sequence number ("PSN") to each packet. As shown, the number of bits increases from left to right. Receiver entity B can confirm that it has received a packet within the sliding window by communicating to sender entity A the PSN it received in an acknowledgment packet within the sliding window. In this regard, a sequence number bitmap can be provided at both sender entity A and receiver entity B. Each bit of the sequence number bitmap represents a packet within the sliding window at the entity. For example, for the TX sliding window 410, if the transmitted packet has been acknowledged, the bit is set to 1. Otherwise, the bit is 0. Once all packets within the TX sliding window 410 are received and acknowledged, the sender entity A can move the sliding window 410 forward to the next set of packets to be transmitted. Once the basic sequence number packet is acknowledged, the sliding window moves forward, therefore, referring to Figure 4 In the example in , once the leftmost 0 is marked, the sliding window will move by one, once the second leftmost 0 is marked, the sliding window will move by another, and once the third 0 is marked, it will move by 3 (because the last two bits are already set).
[0058] The PSN for the sender entity may include a base sequence number ("BSN") and a next sequence number ("NSN"). As shown, the BSN is the PSN value of the oldest packet that has not yet been acknowledged by the receiver entity B. Additionally, as shown, the NSN is the PSN value that should be assigned to the next packet transmitted to the receiver entity B over the connection. For example, when a packet is received from the ULP 310 for transmission, the current PSN may be updated to the NSN. Then, when transmitting packets over the connection, for example, the NSN may be updated to the NSN by NSN = (NSN+1) mod 2 32 , incrementing NSN. Thus, within sliding window 410, bit 0 represents the PSN value of BSN, and bit n represents the PSN value of (BSN + n).
[0059] Although not shown, the receiver entity may also maintain one or more sliding windows. For example, receiver entity B may maintain an RX sliding window for received packets, where each bit represents a packet received through the sliding window. If the packet has been received by receiver entity B, the bit is set to 1. Otherwise, the bit is 0. Receiver entity B may also use PSN to track received packets. For example, BSN may be the PSN value of the oldest packet that the receiver entity has not yet received. When a packet with a PSN value of BSN is received, for example, BSN = (BSN+1) mod 2 may be used. 32 , updating the BSN to the next lowest PSN of the packets not yet received. The updating of the BSN may clear the bits in the sequence number bitmap corresponding to the packets from the previous BSN to PSN. Thus, within the RX sliding window for receiver entity B, bit 0 represents the PSN value of BSN, and bit n represents the PSN value of (BSN+n). Because sender entity A does not acknowledge the acknowledgment sent by receiver entity B, i.e., does not use the PSN for acknowledging packets, receiver entity B does not need to reserve the TX sliding window for the acknowledgment it sends.
[0060] The sender entity and the receiver entity may process packets and corresponding acknowledgments according to a set of rules. For example, if the receiver BSN in the received packet is less than the sender entity's BSN, the sender entity discards the ACK information; otherwise, the sender entity updates its BSN to match the receiver entity's BSN. The sender entity, after adjusting its BSN, applies an OR operation to the receiver entity's sequence number bitmap in the ACK packet with its own sequence number bitmap. After the packet is transmitted, it is buffered by the sender entity until it is acknowledged by the receiver entity. With respect to retransmission of failed packets, the sender entity may be configured to release resources allocated to all ACK packets in the retransmission buffer. In addition, upon expiration of each packet retransmission timer, the sender entity retransmits the packet with the same PSN as the original packet and increments the retransmission counter for the packet.
[0061] The receiver entity may also implement a number of rules. For example, if the PSN value of the received packet is less than the BSN of the received packet, the receiver entity will discard the packet and send an ACK packet with the current BSN. If the PSN value falls within the receiver entity's sliding window, the receiver entity updates the sequence number bitmap by setting the bit at position (PSN-BSN) to 1. If the bit at position (PSN-BSN) is already 1, the packet is discarded; otherwise, the packet is passed to the receiver entity's ULP and the cumulative ACK counter is incremented. If the PSN of the received packet is equal to the BSN of the received packet, the receiver entity updates the BSN to be equal to the next highest PSN that has not yet been received.
[0062] Note that because packets are tracked according to the bitmap, the sliding window is configured to allow an entity to track packets that are received and / or acknowledged out of order within the corresponding sliding window. Thus, as shown, although the packet represented by bits 3 and 4 may be sent by entity A before the packet represented by bits 0, 1, 2, the packet represented by bits 3 and 4 may be received and / or acknowledged before the packet represented by bits 0, 1, 2 in the TX sliding window 410.
[0063] Network congestion can be detected by monitoring packet retransmissions and / or packet round trip delays. To perform congestion control, the size of one or more sliding windows can be adjusted. For example, if congestion is high, it may take longer for entity B to receive and / or acknowledge all packets within the TX sliding window 410. Thus, to reduce congestion, the number of outstanding packets in the network can be reduced by reducing the size of the sliding window 410. In addition to or as an alternative to changing the size of the sliding window, a retransmission timer expiration value responsive to the network congestion state can be adjusted. For example, a lower retransmission frequency can reduce network congestion.
[0064] Figure 3 The communication protocol system 300 can support various transactions, including pull and push transactions. Figure 3 The communication protocol system 300 may be configured to perform transactions using an initiator-target approach, where an "initiator" is an entity that requests a transaction and a "target" is an entity that responds to the request. Such a transaction may involve multiple packets to be transmitted between the initiator and target entities, and thus, the initiator and target entities may be the sender and receiver of packets in the transaction, and TX and / or RX sliding windows may be used to track packets and / or acknowledgments, as described in reference to Figure 4 described. Figure 5 shows an exemplary timing diagram of a pull transaction according to aspects of the present technique, and Figure 6 and 7 An exemplary timing diagram of a push transaction in accordance with aspects of the present technique is shown. Figure 5-7 An exemplary timing diagram may be formed by two entities in a network such as Figure 1 Entities A and B on the connection 110, for example, are Figure 2 The processors 220 and 270 are implemented.
[0065] refer to Figure 5, a timing diagram 500 for an exemplary pull transaction is shown. A pull transaction may be used to "pull" incoming data packets from other entities, such as for a read operation. As shown, a pull transaction is performed through various communication protocol layers of both the initiator entity and the target entity. For example, entity A may be the initiator entity, and the initiator ULP 510 and the initiator RT 530 may be configured as Figure 3 The communication protocol layer of the upper layer protocol 310 and the reliable transport protocol layer 330, and the entity B may be the target entity and the target ULP 520 and the target RT 540 may be configured as Figure 3 The communication protocol layer includes an upper layer protocol 320 and a reliable transport protocol layer 340 .
[0066] As shown, a pull request (“pullReq”) originates from an initiator entity A, such as an initiator ULP 510, which is sent to an initiator RT 530. The initiator entity A may send the pullReq to a target entity B, such as from which it requests incoming data via connection 110. This may be performed by the respective RTs, and thus, the initiator RT 530 is shown sending the pullReq to the target RT 540. Once the pullReq is received by the target entity B, the target RT 540 then sends the pullReq to the target ULP 520 to request permission. The target ULP 520 may then send an acknowledgment message (“ULP-ACK”) to the target RT 540 acknowledging the pullReq, and a pull response (“pullResp”) indicating that the target RT 540 pulls the requested data. In response to the pullResp, the target RT 540 may pull the requested data (“pullData”) and send the pulled data to the initiator RT 530, such as via connection 110. Once the initiator RT 530 receives the requested data, the initiator RT 530 may send a pullResp to the initiator ULP 510 so that the initiator ULP 510 may place or store the received data packet.
[0067] As reference Figure 1 As described above, the initiator entity A and the target entity B may communicate with each other by transmitting packets. Therefore, pullReq and pullData may be packets transmitted through the connection 110 and delivered by the corresponding RT, respectively. Figure 4As described above, reliable transport packets can be tracked by a sliding window. Thus, the pullReq packet can be part of the request TX sliding window maintained by entity A (indicated by the dashed line), and the pullData packet can be part of the data TX sliding window maintained by entity B (indicated by the dot-dash line). Each of these packets can be a reliable transport packet that requires confirmation. For example, although not shown, entity B can send an ACK to entity A of the PSN that references the pullReq packet, and entity A can track the pullReq packet using the request TX sliding window. Similarly, entity A can send an ACK to entity B of the PSN that references the pullData packet, and entity B can track the pullData packet using the data TX sliding window.
[0068] like Figure 5 As shown, the pull transaction allows the initiator entity to perform end-to-end flow control. In particular, the initiator ULP can limit the number and / or total amount of unfinished pull data by rate limiting the pullReq packet to perform end-to-end flow control, which can prevent congestion caused by in-casting in the network. In this way, in-casting congestion can be avoided. For example, the initiator ULP can be configured to request incoming data packets based on scheduling that does not request data packets from multiple entities at the same time. In addition, the initiator RT can be configured to perform admission control to limit the total amount of unfinished pull data.
[0069] refer to Figure 6 , a timing diagram 600 of an exemplary actively requested push transaction is shown. A push transaction may be used to "push" outgoing data packets to other entities, such as for write operations. As shown, a push transaction is performed through various communication protocol layers of both the initiator entity and the target entity. For example, entity A may be the initiator entity, and the initiator ULP 610 and the initiator RT 630 may be configured as Figure 3 The upper layer protocol 310 and the reliable transport protocol layer 330 of the communication protocol layer, and the entity B may be the target entity and the target ULP 620 and the target RT 640 may be configured as Figure 3 The communication protocol layer includes an upper layer protocol 320 and a reliable transport protocol layer 340 .
[0070] As shown, a push request (“pushReq”) may originate from an initiator entity A, such as an initiator ULP 610, which is sent to an initiator RT 630. The initiator entity A may then push the actively requested data to the target entity B, such as via connection 110. This may be performed by the respective RTs, whereby the initiator RT 630 is shown pushing the actively requested data (“pushUnslctdData”) to the target RT 640. The data is actively requested because the target entity B did not request the data. Once the target entity B receives the data, the target RT 640 may request that the received data be placed or stored at the target entity B, and does so by sending a pushReq to the target ULP 620. In response, the target ULP 620 may place or store the received data, and then send an acknowledgement message ULP-ACK to the target RT 640, which acknowledges that the received data has been placed or stored according to the pushReq. For reliable transmission, the target entity B sends an acknowledgement message ("ACK") to inform the initiator entity A that the pushed data has been received and placed, for example, via the connection 110. This is performed by the corresponding RT, whereby as shown, the target RT 640 sends the ACK message to the initiator RT 630. Once the initiator RT 630 receives the ACK message, the initiator RT 630 may send a push completion message ("pushCmpl") to the initiator ULP 610 to inform that the data packet has been received and placed by the target entity.
[0071] As reference Figure 1 As described above, the initiator entity A and the target entity B may communicate with each other by transmitting packets. Therefore, pushUnslctdData and ACK may be packets transmitted via connection 110, respectively. Further, as shown in reference Figure 4 As described above, reliable transmission packets can be tracked by a sliding window. In this way, the pushUnslctdData packet can be part of the data TX sliding window maintained by entity A (indicated by the dotted line). For reliable transmission, the ACK packet sent by entity B can reference the PSN of the pushUnslctdData, which entity A can track using the data TX sliding window. Acknowledgement messages (such as ACK packets) (indicated by the dotted line) are not reliable transmission packets and therefore may not be part of any sliding window at the sender entity B. However, since acknowledgments are cumulative, that is, an ACK received for a packet will indicate that ACKs must have been sent for all previous packets within the sliding window - it is not necessary to make the ACK message reliable.
[0072] like Figure 6As shown, the actively requested push transaction allows the initiator entity to be informed that not only a push data packet is received but also that it has been placed or stored by the target entity. In some cases, the initiator entity may wait until the target entity receives and places the push data packet before pushing another data packet. PushUnslctdData provides lower transaction latency, however, if multiple initiators are sending to the same target, multicasting may still occur in the network. This multicasting risk can be mitigated by limiting the request size supported by pushUnslctdData.
[0073] and Figure 6 on the contrary, Figure 7 A timing diagram 700 for a solicited push transaction is shown. As shown, the push transaction is performed by various communication protocol layers of both the initiator entity and the target entity. For example, entity A may be the initiator entity, and the initiator ULP 710 and the initiator RT 730 may be configured as Figure 3 The upper layer protocol 310 and the reliable transport protocol layer 330 of the communication protocol layer, and the entity B may be the target entity, and the target ULP 720 and the target RT 740 may be configured as Figure 3 The communication protocol layer of the upper layer 320 and the reliable transport protocol layer 340.
[0074] and Figure 6 resemblance, Figure 7 It is shown that the pushReq can originate from the initiator entity A at the initiator ULP 710, which can be sent to the initiator RT 730. However, compared to the timing diagram 600, at this time, the initiator RT 730 will not send the packet to be pushed to the target entity B. However, for example, through the connection 110, only the request is sent to the target entity B, and the request may or may not be authorized by the target entity B. This request and authorization process or "request" process can be performed by the corresponding RT, for example, it can be performed by its corresponding request sublayer. Therefore, the initiator RT 730 is shown as sending a push request ("pushSlctdReq") to the target RT 740, and the target RT 740 can decide whether and / or when to authorize the pushSlctdReq. From the perspective of the request, the target pushGnt is similar to the initiator pullReq. For example, entity B can limit the total number of outstanding authorized pushSlctdData to prevent multicasting to entity B, causing network congestion. If and when the target RT 740 authorizes the request, the target RT 740 may send a push grant (“pushGrnt”) back to the initiator RT 730 .
[0075] Once the initiator entity A receives the pushGrnt, the initiator entity A may push the requested data to the target entity B, for example, via connection 110. This may be performed by the corresponding RT, so the initiator RT 730 is shown pushing the requested data to the target RT 740. Figure 6 In contrast to the actively requested data pushed in , the data here is effectively requested by the pushGrnt from the target RT 740. Once the pushGrnt is received by the initiator RT 730, the rest of the timing diagram is similar to Figure 6 . Therefore, the target RT 740 may send a pushReq to the target ULP 720 to place or store the received data at the target entity B. In response, the target ULP 720 may place or store the received data and then send a ULP-ACK message to the target RT 740 to confirm that the received data has been placed according to the pushReq. For reliable transmission, the target entity B sends an ACK message through the target RT 740 to inform the initiator entity A that the pushed data has been received and placed, for example, through the connection 110 to the initiator RT 730. Once the initiator RT 730 receives the ACK message, the initiator RT 730 may send a pushCmpl message to the initiator ULP 710 to inform that the data packet has been received and placed by the target entity B.
[0076] As reference Figure 1 As described above, the initiator entity A and the target entity B may communicate with each other by transmitting packets. Therefore, pushSlctdReq, pushGrnt, pushSlctdData and ACK may be packets transmitted via connection 110, respectively. Further, as shown in reference Figure 4As described, reliable transmission packets can be tracked by a sliding window. Thus, pushSlctdReq can be part of the request TX sliding window maintained by entity A (indicated by the dashed line), the pushGrnt packet can be part of the data TX sliding window maintained by entity B (indicated by the dotted line), and the pushSlctdData packet can be part of the data TX sliding window maintained by entity A (indicated by the dotted line). For reliable transmission, the ACK packet sent by entity B can reference the PSN of pushSlctdData, which entity A can track using the data TX sliding window. Similarly, although not shown, entity A can send an ACK for the pushGrnt packet, which entity B can track using its data TX sliding window, and entity B can send an ACK for pushSlctdReq, which entity A can track using its request TX sliding window. However, confirmation messages (such as the ACK packet shown (indicated by the dashed line)) are not reliable transmission packets and therefore may not be part of any sliding window at the sender entity B.
[0077] like Figure 7 As shown, a solicited push transaction allows an initiator entity to request authorization to send data from a target entity before actually sending the data. In this way, the target entity can have control over the incoming data, which is particularly useful when multiple initiator entities are trying to push data to the target entity, and if the pushed data is large or if the network is congested. Since incast congestion can be caused by packets not being delivered to the recipient as fast as they were transmitted and / or by multiple entities trying to send packets to the same entity at the same time, this solicitation process can reduce incast congestion.
[0078] In some cases, the communication protocol system can be configured to perform Figure 6 and 7One of the push transactions shown, while in other cases, the communication protocol system can be configured to perform two push transactions. In the case where the communication protocol system is configured to perform two push transactions, the system can be configured to determine which push transaction to use based on one or more factors. For example, the initiator RT can determine whether the push request should be sent as a solicited request or an unsolicited request. The initiator RT can determine whether to send a solicited push request or an unsolicited push based on the length of the push request from the initiator ULP. As an example, if the push request needs to push a large amount of data, such as meeting a predetermined size threshold, a solicited push request can be used to ensure that the larger request does not cause congestion; otherwise, a solicited push can be used. As another example, whether to use a solicited request or an unsolicited push can be based on network conditions, such as congestion levels, where a solicited request can be used when the congestion meets a predetermined threshold level.
[0079] On the other hand, Figure 3 The communication protocol system 300 can support both ordered and unordered transactions over a connection. In this regard, Figure 8 shows an exemplary timing diagram for ordered transactions over a connection according to aspects of the present technique, and Fig. 9 An exemplary timing diagram for out-of-order transactions over a connection in accordance with aspects of the present technique is shown. Figure 8-9 An exemplary timing diagram may be provided by two entities in a network (such as by Figure 1 The connection 110 entities A and B) for example through Figure 2 The processors 220 and 270 are implemented.
[0080] refer to Figure 8 Various transactions may be performed by various communication protocol layers of both the initiator entity and the target entity, such as those described above with reference to Figure 5-7 For example, entity A may be the initiator entity, and the initiator ULP 810 and the initiator RT 830 may be configured as Figure 3 The upper layer protocol 310 and the reliable transport protocol layer 330 of the communication protocol layer, and the entity B may be the target entity and the target ULP 820 and the target RT 840 may be configured as Figure 3 The communication protocol layer includes an upper layer protocol 320 and a reliable transport protocol layer 340 .
[0081] Referring to the timing diagram 800, a number of requests may originate from the initiator entity A, including pull requests (such as pullReq_1) and push requests (such as pushReq_0, pushReq_2, and pushReq_3). As described above, these requests may be sent by the initiator ULP 810 to the initiator RT 830. Once the initiator RT 830 receives these requests, the initiator RT 830 may optionally determine whether the push request should be sent as solicited or unsolicited, as described above with reference to Figure 7 Thus, in this example, the initiator RT 830 may determine that pushReq_0 and pushReq_2 are to be sent as solicited, while pushReq_3 is to be sent as unsolicited. The initiator RT 830 may then send these pull requests and push requests to the target RT 840, for example, via connection 110.
[0082] Requests may be sent by the initiator ULP 810 in a specific order as indicated by a request sequence number (“RSN”), which may be assigned by the initiator ULP 810. In some cases, the initiator RT 830 may also assign a solicited sequence number (“SSN”) specifically to the solicited push requests, which may be an incremental number as shown. When the requests are sent as packets between two entities, the requests may be assigned a sequence of numbers in ascending order based on the order of the RSNs. Thus, as shown, the requests may be assigned PSNs within one or more TX sliding windows maintained by the initiator entity A based on the RSNs. For example, within the request TX sliding window of entity A (shown by the dashed line pointing to B), pushSlctdReq_0 is assigned PSN = 0, pullReq_1 is assigned PSN = 1, and pushSlctdReq_2 is assigned PSN = 2. Note that since pushReq_3 from the initiator ULP 810 does not need to be assigned as shown, Figure 5 The request shown, therefore, has no corresponding pushUnslctdReq sent between the RTs. Although the ULP may know the RSN and SSN, the ULP may not know the PSN and can only be used by the RTs in the packet.
[0083] In response to the solicited push request, the target RT 840 may send push grants to the initiator RT 830, such as pushGnt_0 and pushGnt_2, in the order in which the requests were received. The push grants may be assigned PSNs in ascending order within one or more TX sliding windows maintained by the target entity B, in the same order as the RSNs of the push requests. For example, within the data TX sliding window of entity B (shown by the dotted line pointing to A), pushGrnt_0 is assigned PSN = 1000, and pushGrnt_2 is assigned PSN = 1001. However, the initiator RT 830 may not receive the push grants in the same order as the transmission order of the push requests. Therefore, as shown, the initiator RT 830 receives pushGrnt_2 before pushGrnt_0.
[0084] Nevertheless, the initiator RT 830 can determine the correct order of push authorization based on their respective PSNs and push data packets based on that order. In this way, although the initiator RT 830 receives pushGrnt_2 before pushGrnt_0, the initiator RT 830 can first push the data requested by pushGrnt_0 using pushSlctdData_0, and then push the data requested by pushGrnt_2 to the target RT 840 using pushSlctdData_2. PSNs are also assigned to the pushed data packets in ascending order within one or more TX sliding windows maintained by the initiator entity A according to the transmission order. For example, within the data TX sliding window of entity A (represented by the dotted line pointing to B), pushSlctdData_0 is assigned PSN = 200, and pushSlctdData_2 is assigned PSN = 201. Note that pushReq_3 does not require authorization, as shown in reference Figure 6 As described above, therefore, as indicated by the curved arrow, it jumps directly to pushUnslctdData_3, which pushes the actively requested data. In this example, pushUnslctdData_3 is also assigned PSN = 202 in the data TX sliding window of entity A.
[0085] The target RT 840 receives the request and then sends a corresponding request to the target ULP 820 in the order of ULP-Req-0-1-2-3, which is the same order as the transmission of the request from the initiator ULP 810 shown at the top of the timing diagram 800. Figure 5-7As described above, these ULP-Reqs require the target ULP 820 to grant permission to pull data or place pushed data at the target entity B. Note that the pull request pullReq_1 does not require authorization, as shown in Figure 5 As described above, therefore, as shown by the curved arrow, jump directly to ULP-Req. In response to ULP-Req, as shown above with reference Figure 5-7 As described above, the target ULP 820 may send an acknowledgement ULP-ACK to the target RT 840. In this ordered system, the ULP-ACKs are sent in the order of ULP-ACK-0-1-2-3, which is the same order as the transmission order of the request from the initiator ULP 810.
[0086] After the ULP-ACK, for the push transaction, an ACK (or data acknowledgment) of the confirmation data packet is then sent by the target RT 840 to the initiator RT 830 to notify the safe receipt and placement of the reliable transport data packets. As an example, entity B sends ACK-eBSN=3,203 to notify entity A that all request packets up to PSN=3 and all data packets up to PSN=203 have been received and placed. Once the ACK is received, the initiator RT 830 can send a completion message pushCompl_0 to the initiator ULP 810. In addition, in some cases, confirmation packets can be appropriately piggybacked on other reliable transport packets. For example, the requests pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 are reliable transport packets that require ACK, but the acknowledgments (or request ACKs) for these requests are not explicitly shown in the timing diagram 800 because they may be piggybacked on reliable transport packets such as pushGrnt_0 and pushGrnt_2.
[0087] Also after the ULP-ACK, a pull request may also be responded to. Thus, as shown, the target ULP 820 may send pullResp_1, which instructs the target RT 840 to pull the requested data. The target RT 840 then sends the pulled data to the initiator RT 830 using pullData_1. In this example, pullData_1 is assigned PSN = 1002 within the same data TX sliding window of entity B as pushGrnts (as shown by the dotted line pointing to A). The initiator RT 830 then sends pullResp_1 to the initiator ULP 810 so that the initiator ULP 810 can place or store the received data packet at entity A. After the data packet is placed or stored at entity A, an acknowledgment may be sent to notify entity B of safe reception. Thus, as shown, entity A sends ACK-eBSN = 3,203 to notify the entity that the pullData_1 packet has been safely received.
[0088] In this ordered system, the order of completion messages received by the initiator ULP 810 near the bottom of the timing diagram 800 is the same as the order of requests sent by the initiator ULP 810 near the top of the timing diagram 800. The order is maintained on both the ULPs of the initiator and target entities, where the target RT makes requests to the target ULP in the same order that the initiator ULP sent the requests to the initiator RT. The ordered system ensures that requests are delivered once and only once over the connection. In contrast, there may be no ordering requirements between transactions conducted in different directions over the connection.
[0089] Fig. 9 An exemplary timing diagram 900 for out-of-order transactions over a connection according to aspects of the present technique is shown. The timing diagram 900 may be performed by various communication protocol layers of both an initiator entity and a target entity. For example, entity A may be an initiator and an initiator ULP 910 and an initiator RT 930 may be configured as Figure 3 The high-level protocol 310 and the reliable transport protocol layer 330 of the communication protocol layer, and the entity B may be the target entity and the target ULP 920 and the target RT 940 may be configured as Figure 3 The communication protocol layer includes an upper layer protocol 320 and a reliable transport protocol layer 340 .
[0090] Referring to timing diagram 900, many of the same transactions as timing diagram 800 are shown and labeled similarly. For example, once various requests are sent from initiator ULP 910 to initiator RT 930, similar to timing diagram 800, the requests are then transmitted to target RT 940. For solicited push requests (such as pushSlctdReq_0 and pushSlctdReq_2), push grants (such as pushGrnt_0 and pushGrnt_2) are sent by target RT 940 to initiator RT 930. In response to the push grants, the solicited data is pushed by initiator RT 930 to target RT 940, as shown by pushSlctdData_0 and pushSlctdData_2. For active requests such as pushUnslctdReq_3, initiator RT 930 can push the actively requested data to target RT 940, as shown by pushUnslctdData_3. For a pull request such as pullReq_1, the initiator RT 930 may send the pull request to the target RT 940 and then to the target ULP 920. The target ULP 920 may then respond with pullResp_1, which responds to the initiator RT 930 with pullData_1. The initiator RT 930 then sends pullResp_1 to the initiator ULP 910, which handles placing and / or storing the pulled data at entity A. As described above with reference to Figure 8 As described above, these groups can also be tracked by a sliding window.
[0091] The timing diagram 900 also shows several aspects in which the out-of-order system differs from the ordered system of the timing diagram 800. One difference is that the PSNs for the packets are assigned based on the transmission order rather than based on the corresponding RSNs. For example, PSNs are assigned to pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 based on the transmission order of the initiator RT 930, which is in the same order as the corresponding RSNs. Due to the transmission order, the pushGrnt_0 and pushGrnt_2 packets also have PSNs in the order of the corresponding RSNs. However, the pushUnslctdData_3, pushSlctdData_0, and pushSlctdData_2 packets are transmitted in an order different from the order of the corresponding RSNs, thereby generating PSNs that are not in the same order as the corresponding RSNs. In this way, pushUnslctdData_3 with PSN = 200 is smaller than pushSlctdData_0 with PSN = 201 and pushSlctdData_2 with PSN = 202.
[0092] Another difference is that the target entity may process transactions out of order. Therefore, as shown, the ULP-Req and ULP-ACK in the timing diagram 900 are not all sent after the various requests and authorizations are passed around the corresponding RTs. Instead, once the requests and / or authorizations between RTs are completed with respect to a particular transaction, the ULP-Req and ULP-ACK corresponding to the particular transaction are sent. For example, with respect to the transaction pullReq_1, once pullReq_1 is transmitted from the initiator RT 930 to the target RT 940, ULP-Req_1 and ULP-ACK-1 are transmitted. As another example with respect to the transaction pushUnslctdReq_3, once pushUnslctdData_3 is transmitted from the initiator RT 930 to the target RT 940, ULP-Req_3 and ULP-ACK-3 are transmitted. With respect to transactions pushSlctdReq_0 and pushSlctdReq_2, ULP-Reqs and ULP-ACK are sent later after the push grant and solicited data packets are pushed from the initiator RT 930 to the target RT 940. In addition, since pushSlctdData_0 is received at the target RT 940 after pushSlctdData_2, the ULP-Reqs and ULP-ACKs of these two are sent in reverse order. Further, although ULP-Req-1 and ULP-ACK-1 may be transmitted before ULP-Req_3, ULP-ACK_3, ULP-Req-0, ULP-Req-2, ULP-ACK-0, and ULP-ACK-UL, the target ULP 920 may still process a pull request after a push request.
[0093] As a result of the out-of-order processing of transactions by the target entity, confirmation and completion messages may therefore be sent out-of-order to the initiator entity. In this way, the initiator may also process transactions out of order. Therefore, as shown, the target RT 940 sends an ACK for pushUnslctdData_3, which prompts the initiator RT 930 to send pushCompl_3 to the initiator ULP 910 before the ACKs for pushSlctdData_0 and pushCompl_0, before pushCompl_2, and before pullResp_1. Further as shown, because the push solicited in this example is completed before the pull request, the ACK for pushSlctdData_0, pushComp_0, and pushComp_2 may be transmitted before the pullResp_1 from the initiator RT 930.
[0094] like Fig. 9As shown, out-of-order processing of transactions by entities allows certain transactions that are processed faster to proceed before slower transactions on a given connection, which can improve the overall efficiency of the connection. For example, by allowing out-of-order transmission of ULP-Req and ULP-ACK, this allows transactions with fewer steps (such as pulls and actively requested pushes) to proceed at the target ULP without waiting for other transactions with more steps (such as requested pushes) at the RT to complete the request process. As another example, the pull request in this example can be processed later by the target ULP 920 because the push transaction has a faster processing time. Additionally or alternatively, the network can deliver packets out of order. Out-of-order processing of transactions can therefore provide greater flexibility to speed up the entire system and increase throughput.
[0095] On the other hand, Figure 3 The communication protocol system 300 can provide effective error handling, thereby reducing the chance of deadlock. In this regard, Fig.10 shows an exemplary timing diagram for a receiver not ready ("RNR") negative acknowledgement ("NACK") in accordance with aspects of the present technique, and Fig.11 An exemplary timing diagram for error completion ("compl-in-error") in accordance with aspects of the present technique is shown. Figure 10-11 An exemplary timing diagram may be formed by two entities in a network such as Figure 1 The connection 110 between entities A and B) is for example through Figure 2 The processors 220 and 270 are implemented.
[0096] refer to Fig.10 Various transactions may be performed by various communication protocol layers of both the initiator entity and the target entity, such as those described above with reference to Figure 5-9 For example, entity A may be the initiator entity, and the initiator ULP 1010 and the initiator RT 1030 may be configured as Figure 3 The upper layer protocol 310 and the reliable transport protocol layer 330 of the communication protocol layer, and the entity B may be the target entity and the target ULP 1020 and the target RT 1040 may be configured as Figure 3 The communication protocol layer includes an upper layer protocol 320 and a reliable transport protocol layer 340 .
[0097] Referring to timing diagram 1000, many of the same transactions as timing diagram 800 are shown and labeled similarly. For example, timing diagram 1000 shows pushReq_0, pullReq_1, and pushReq_2 originating from initiator ULP 1010. Once the various requests are transmitted from initiator ULP 1010 to initiator RT 1030, these requests are transmitted to target RT 1040, similar to timing diagram 800. For the solicited push requests pushSlctdReq_0 and pushSlctdReq_2, push grants pushGrnt_0 and pushGrnt_2 are sent by target RT 1040 to initiator RT 1030. In response to the push grants, the solicited data is pushed by initiator RT 1030 to target RT 1040, as shown by pushSlctdData_0 and pushSlctdData_2. In addition, a pull request pullReq_1 is sent by the initiator RT 1030 to the target RT 1040. Then, the target RT 1040 sends pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 to the target ULP 1020. The target RT 1040 also sends an acknowledgment of the request to the initiator RT 1030 using ACK-0-1-2.
[0098] However, in timing diagram 1000, the target entity is not yet ready for push and pull requests. Therefore, the target ULP 1020 sends a negative acknowledgement ("NACK"), which notifies the target ULP 1020 that it is not yet ready and that the target RT 1040 should try again later. Therefore, as shown, the target ULP 1020 sends pushNACK_0-retry in response to pushSlctdReq_1, sends pullNACK_1-retry in response to pullReq_1, and sends pushNACK_2-retry in response to pushSlctdReq_2. These NACKs may include a reason for the NACK, which in this example is that the target entity is not yet ready, and / or may include a new timer expiration value for retransmission. The target RT 1040 then sends a corresponding NACK to the initiator RT 1030, as shown by Nush-200 and NACK-201 that reference the PSNs of the pushSlctdData_0 and pushSlctdData_2 packets. Note that the initiator does not have a NACK for pullReq_1 because ACK-0-1-2 has already acknowledged it. Instead, the target RT 1040 tracks the pullNACK-1-retry and will pass it to the target ULP 1020 again when the timer expires. Note that the NACK has the same eBSN because the sliding window is stuck due to the inability to acknowledge the sender. However, although NACK-201 arrives at the initiator RT 1030, the NACK-200 packet may be dropped in the network and not reach the initiator RT 1030 due to any of a number of reasons, and as previously mentioned, ACK and NACK messages cannot be transmitted over reliable transport.
[0099] Because the initiator RT 1030 receives a NACK for pullSlctdReq_2, the initiator RT does not attempt to retransmit pushSlctdData_2. On the contrary, since the initiator RT 1030 did not receive a NACK for pushSlctdReq_0, the initiator RT 1030 attempts to retransmit through pushData_0-retry. For example, the initiator RT 1030 can be configured to attempt to retransmit using the same PSN if an ACK for the pushed data is not received within a predetermined time period. Once the target RT 1040 receives pushData_0-retry, the target RT 1040 recognizes that pushData_0-retry is a retransmission of the data packet, for example, based on the fact that the PSN of pushData_0-retry is the same as the previously received pushSlctdData_0. Then, the target RT 1040 sends ULP-pushReq_0 to the target ULP 1020. This time, the target ULP 1020 is ready and responds with ULP-ACK-0.
[0100] Then, after a period of time, the target RT 1040 tries again with ULP-pullReq_1 and ULP-pushSlctdReq_2 to the target ULP 1020. As shown, this time the target ULP 1020 is ready and completes the request. The rest of the timing diagram 1000 is similar to the timing diagram 800, where the target ULP 1020 sends confirmations ULP-ACK-0, ULP-ACK-1, and ULP-ACK-2 to the target RT 1040, and then the target RT 1040 sends confirmations ACKs-200-201 to the initiator RT 1030, which in turn sends completion messages pushComp_0 and pushCompl_2. For a pull request, the target ULP 1020 sends pullResp_1 to the target RT 1040 , which then sends pullResp_1 to the initiator RT 1030 , and then sends a completion message pullCompl_1 to the initiator ULP 1010 .
[0101] therefore, Fig.10 It is shown that by allowing the initiator RT to retransmit requests and / or data when an ACK or NACK is not received within a predetermined time period, the RT operates robustly and reliably on a potentially lossy network. In addition, by not allowing the initiator RT to retransmit when a NACK is received, congestion on the network can be reduced. Still further, by allowing the target RT to retransmit requests and / or data to the target ULP, the request can be processed at a later time when the target ULP is ready without causing deadlock or timeouts.
[0102] refer to Fig.11 , various transactions may be performed through various communication protocol layers of both the initiator entity and the target entity, such as those described above with reference to Figure 5-9 For example, entity A may be the initiator and the initiator ULP 1110 and the initiator RT 1130 may be configured as Figure 3 The communication protocol layer of the upper layer protocol 310 and the reliable transport protocol layer 330, and the entity B may be the target entity and the target ULP 1120 and the target RT 1140 may be configured as Figure 3 The communication protocol layer includes an upper layer protocol 320 and a reliable transport protocol layer 340 .
[0103] Referring to timing diagram 1100, many of the same transactions as timing diagram 800 are shown and labeled similarly. For example, timing diagram 1100 shows pushReq_0, pullReq_1, and pushReq_2 originating from initiator ULP 1110. Once the various requests are sent from initiator ULP 1110 to initiator RT 1130, the requests are transmitted to target RT 1140 similar to timing diagram 800. For the solicited push requests pushSlctdReq_0 and pushSlctdReq_2, push grants pushGrnt_0 and pushGrnt_2 are sent by target RT 1140 to initiator RT 1130. In response to the push grants, initiator RT 1130 pushes the solicited data to target RT 1140, as shown with pushSlctdData_0 and pushSlctdData_2. The initiator RT 1130 sends a pull request pullReq_1 to the target RT 1140. Then, the target RT 1140 sends pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 to the target ULP 1120.
[0104] However, in timing diagram 1100, the target ULP 1120 completes pushSlctdData_0 incorrectly due to any of a variety of reasons. Therefore, the target ULP 1120 sends pushNACK_0-compl-in-err, which informs the target RT 1140 of the error. The NACK may include a reason for the NACK, which in this example is that the placement or storage of the push data was completed incorrectly. The NACK may optionally include information about the error, such as the cause. The target RT 1140 then sends a corresponding NACK to the initiator RT 1130, as shown by NACK-200 that references the PSN of pushSlctdData_0. However, due to any of a variety of reasons, NACK-200 cannot reach the initiator RT 1130. Furthermore, as shown, the target ULP 1120 successfully completes the other requests and sends ULP-ACK_1, ULP-ACK_2, and pullResp_1 to the target RT 1140, prompting the target RT 1140 to send ACK-201 and pullResp_1 to the initiator RT 1130.
[0105] Because the initiator RT 1130 did not receive a NACK for pushSlctdReq_0, the initiator RT 1130 attempts to retransmit using pushData_0-retry. For example, the initiator RT 1130 can be configured to attempt to retransmit using the same PSN if an ACK for the push data is not received within a predetermined time period. Once the target RT 1140 receives pushData_0-retry, the target RT 1140 recognizes that pushData_0-retry is a retransmission of the data packet, for example, based on the fact that the PSN of pushData_0-retry is the same as the previously received pushSlctdData_0. In this way, the target RT 1140 resends NACK-200 without having to send another push request to the target ULP 1120.
[0106] In addition, in response to an erroneous complete negative acknowledgement, a resynchronization may be performed. For example, as shown, a resynchronization may be initiated by the initiator RT 1130 using "resync-pkt". The resynchronization prompts the target RT 1140 to send an acknowledgement that allows one or more current sliding windows to move to the next set of packets to be transmitted and / or received. As shown, this is accomplished by the target RT 1140 sending an acknowledgement ACK-200 instead of a negative acknowledgement to the initiator RT 1130 in response to the resync-pkt. The initiator RT 1130 then sends completion messages pushCompl_0-in-err, pullCompl_1, and pushCompl_2 to the initiator ULP 1110, which notify successful completion and error completion.
[0107] Fig.11 The resynchronization process is shown to prevent the current sliding window from getting stuck. Furthermore, the resynchronization process allows a more resilient connection to be maintained between entities, in contrast to many protocols where complete error completion may result in a connection timeout or disconnection. Fig.11 An exemplary time scale for the process is also shown. As shown, one round trip time ("RTT") may pass between the time pushSlctdReq_0 is sent by the initiator RT 1130 and the time pushGrnt_0 is received by the initiator RT 1130. Then, multiple RTTs may have passed between the time pushGrnt_0 is received by the initiator RT 1130 and the time NACK-200 is being received by the initiator RT 1130. Another RTT may pass between the time resync-pkt is sent by the initiator RT 1130 and the time pushCompl_0-in-err is sent by the initiator RT 1130.
[0108] Although providing an ordered system Fig.10 and 11 , but in other examples, the timing diagram can be executed by an out-of-order system. In this case, a similar process can be performed without the in-order requirement.
[0109] return Figure 3, the communication protocol system 300 may be additionally provided with or associated with one or more congestion control engines. Such a congestion control engine may be configured with any of a variety of algorithms, such as SWIFT, BBR, GCN, etc. In this regard, the congestion control algorithm may be implemented in software, firmware, or hardware. For example, the congestion control algorithm may be implemented in host software, network interface ("NIC") firmware, or a hardware rate update engine. For example, the communication protocol system 300 may provide a rate update event and result queue interface to the rate update engine. A congestion signal may be provided in an event queue, which may include a measurement of RTT, an explicit congestion notification ("ECN") marker, a retransmission status, and the like.
[0110] The rate update engine may report the results back to the communication protocol system 300, based on which congestion control may be implemented. For example, the report may include a congestion window ("Cwnd"), which is the total number of outstanding TX packets. When this value is between 0 and 1, the communication protocol system 300, e.g., the sender RT, may apply an additional inter-packet gap to limit the number of packet transmissions to less than 1 per RTT. As another example, the report may include a retransmission timeout ("RTO"), which is the time the sender entity waits before retransmitting a pending TX packet if an ACK is not received.
[0111] Figures 12A-17B 2 shows exemplary headers and definitions according to aspects of the present technology. These example headers are provided as examples only and may in fact take any other format and may include alternative or additional information. Figure 4 Sliding windows and / or Figure 5-11 When describing the timing diagram of FIG. 1 , various communication protocol layers of the entities may use exemplary headers. For example, Fig. 12A -B describes an exemplary basic header that will be included by all packets within the communication protocol system. Fig.13A -F describes exemplary headers for request and authorization packets in a communication protocol system. Fig.14A -C describes an exemplary header for a data packet in a communication protocol system. Fig.15A -B describes an exemplary resynchronization packet in a communication protocol system. Fig.16A -B describes an exemplary acknowledgment packet in a communication protocol system. Fig.17A -B describes an exemplary negative acknowledgement packet in a communication protocol system.
[0112] refer to Fig. 12A and 12B, the exemplary basic header shown includes various information fields. For example, the "Proto Type" field can specify various protocols to be used for the packet, such as RT protocol, ULP protocol, storage protocol, etc. "ULP Protocol" can specify different ULP types supported, for example, RDMA, NVMe, etc. The "Packet Type" field can specify the type of packet, such as the various requests, authorizations, data, ACK, NACK, resynchronization, etc. mentioned above. The basic header can include various sequence numbers for the packet, such as the PSN and RSN mentioned above, as well as the expected basic packet sequence number in the RX direction for the data sliding window ("eDBSN"), and the expected basic packet sequence number in the RX direction for the request sliding window ("eRBSN"). Further, the basic header can include a destination CID that identifies the communication direction through a connection to a recipient entity.
[0113] Steering Figures 13A-13F , which shows exemplary request and authorization headers, each including Fig. 12A The basic header and optionally some additional information. For example, Fig.13A and 13D An exemplary pull request header is described that may include additional information, such as a "request length" field that specifies the size of the return pull response packet and / or reserved customizable fields. Fig. 13B and 13E An exemplary push request header is described, which may similarly include a request length field that specifies the size of the outgoing push data packet.For a solicited push request, the push request header may further include an SSN field that specifies the order of the request. Fig. 13C and 13F An exemplary push authorization header for a solicited push request is described, which may include a request length field, an SSN field, an SPH reassembly context ID, and an SPH replay count field.
[0114] Figures 14A-14C Exemplary data headers are shown, each including Fig. 12A The basic header and optionally some additional information. For example, Fig.14A An example pull data header and Fig. 14B The exemplary push data headers in each include Fig. 12A The basic header has no additional information. Instead, Fig. 14C An exemplary push active request data header includes Fig. 12A The basic header of the UDP packet header and additionally includes a request length field that specifies the size of the actively requested data packet being pushed.
[0115] Fig.15A and 15BAn exemplary resynchronization header is described, which includes Fig. 12A The resync header may include a basic header of the ULP and optional additional information. For example, the resync header may include a "ResyncCode" field that specifies the target ULP to which the resync packet is sent, memory release, and other information. The resync header may further include a "resynch type" field that specifies the type of the original packet being resynchronized. If the resync type is a passive push request or push authorization, the SSN may be provided.
[0116] Fig.16A and 16B An exemplary acknowledgement or ACK header is described. Fig.16A , an exemplary ACK header includes some Fig. 12A The ACK header contains the same fields as the basic header of the ACK, but instead of the PSN and RSN fields, the ACK header includes the "Receiver Data Window Sequence Bitmap" and "Receiver Request Window Sequence Bitmap" fields. Figure 4 As described, a value of "1" in such a bitmap indicates that the packet has been acknowledged, and a value of "0" indicates that the packet has not yet been acknowledged. In addition, a value of "N0" in an exemplary ACK header for the field NACK indicates that the packet is an ACK rather than a NACK. Further, the ACK header may include a congestion control metadata field, which the RT may carry between two entities to facilitate congestion control algorithms, and one or more timestamps for measuring latency in the network.
[0117] Fig.17A and 17B An exemplary negative acknowledgement or NACK header is described. Fig.17A , an exemplary NACK header includes some Fig. 12A The exemplary NACK header includes the same fields as the basic header of the ACK header, but instead of the PSN and RSN fields, the NACK header includes a "NACK sequence number" field, which specifies the PSN of the packet being negatively acknowledged. The exemplary NACK header also includes some of the same fields as the ACK header of Figure 16. For example, the value "N1" in the exemplary NACK header for the field NACK indicates that the packet is a NACK. The exemplary NACK header also includes fields that are not found in the basic header or the ACK header. The NACK header may include a "NACK code" field, which may, for example, provide the reason for the NACK, such as excessive network jitter, decreased receiver resources, receiver not ready, etc. The NACK header may also include a "wnd" field indicating the sliding window to which the NACK points. In the case of RNR-NACK, the NACK header may further include an "RNR-NACK timeout" field, which specifies the timer expiration value for retransmission.
[0118] Fig.18An exemplary flow chart illustrating an exemplary method according to aspects of the present disclosure is shown. The method may be performed using any of the above-described systems, variations thereof, or a variety of systems having different configurations. It should be understood that the operations involved in the method described below do not have to be performed in the exact order described. Rather, various operations may be processed in a different order or simultaneously, and operations may be added or omitted. Although Fig.18 An exemplary method is shown, but variations of the method may be performed, for example, as described above with reference to Figure 1-11 described.
[0119] Reference Fig.18 , shows an exemplary flowchart 1800 illustrating a solicited push transaction. Flowchart 1800 may be performed by one or more entities in a network, such as by Figure 1 Any of entities A, B, C executes, for example, Figure 2 The processor 220 executes. Figure 2 As mentioned, the processor 220 may include a processor on one entity (such as the entity 110), or a processor on multiple entities (such as two or more entities among entities A, B, C, etc.). In this way, the processor 220 may receive data and perform reliable transmission, as described above with reference to Figure 1-11 As described, this may include using headers such as Figures 12A-17B shown.
[0120] refer to Fig.18 At block 1810, an initiator entity determines that outgoing data is to be transmitted to a target entity. At block 1820, the initiator entity transmits a solicited push request to the target entity requesting that the outgoing data be placed at the target entity. At block 1830, the initiator entity receives a push authorization from the target entity in response to the solicited push request. At block 1840, the initiator entity transmits the outgoing data to be placed at the target entity to the target entity in response to the push authorization.
[0121] The technology generally involves a communication protocol for reliably transmitting packets over a connection. The technology provides request-based push transactions that provide recipient entity control over incoming data, thereby reducing incast congestion and tail latency. The technology further uses sliding windows and bitmaps to support out-of-order transactions over the connection, which can improve the overall efficiency of processing packets over the connection. The technology further provides handling of failed transmissions, reduces retransmission attempts and uses resynchronization to prevent connection drops, resulting in a more resilient connection.
[0122] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made by way of example, rather than by limiting the subject matter defined by the claims. In addition, the examples described herein and the phrases "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.
Claims
1. A method comprising: transmitting, by a sender entity, a plurality of packets in a first sequence over a connection to a receiver entity; maintaining, by the sender entity, one or more sliding windows comprising a plurality of bits, wherein each bit of the sliding window represents a corresponding packet of the plurality of packets; receiving, by the sender entity, one or more acknowledgments indicating that the receiver entity has received one or more of the plurality of packets, each of the acknowledgments referencing a respective packet of the plurality of packets; and The value of one or more bits of the plurality of bits in the sliding window corresponding to the one or more received acknowledgments is modified by the sender entity.
2. The method according to claim 1, wherein: The acknowledgments are received in a second order different from the first order.
3. The method according to claim 1, further comprising: The size of the one or more sliding windows is adjusted by the sender entity based on one or more congestion parameters.
4. The method according to claim 1, wherein: The plurality of packets include one or more of: a request for a data packet, a data packet, or an acknowledgment.
5. The method according to claim 4, wherein: The one or more sliding windows include a request sliding window.
6. The method according to claim 4, wherein: The one or more sliding windows include a data sliding window.
7. The method according to claim 1, wherein: The plurality of packets include one or more data packets in response to a pull request.
8. The method according to claim 1, wherein: The plurality of packets include at least one push authorization packet in response to a solicited push request.
9. A system comprising: One or more processors configured to: transmitting a plurality of packets in a first order over a connection to a recipient entity; maintaining one or more sliding windows comprising a plurality of bits, wherein each bit of the sliding window represents a corresponding packet from the plurality of packets; receiving one or more acknowledgments indicating that the recipient entity has received one or more of the plurality of packets, each of the acknowledgments referencing a respective packet of the plurality of packets; and The value of one or more bits of the plurality of bits in the sliding window corresponding to the one or more received acknowledgments is modified.
10. The system according to claim 9, wherein: The acknowledgments are received in a second order different from the first order.
11. The system of claim 9, wherein the one or more processors are further configured to: The size of the one or more sliding windows is adjusted based on one or more congestion parameters.
12. The system according to claim 9, wherein: The plurality of packets include one or more of: a request for a data packet, a data packet, or an acknowledgment.
13. The system according to claim 12, wherein: The one or more sliding windows include a request sliding window.
14. The system according to claim 12, wherein: The one or more sliding windows include a data sliding window.
15. The system according to claim 9, wherein: The plurality of packets include one or more data packets in response to a pull request.
16. The system of claim 9, wherein: The plurality of packets include at least one push authorization packet in response to a solicited push request.
17. One or more non-transitory computer-readable storage media encoded with operable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: transmitting a plurality of packets in a first order over a connection to a recipient entity; maintaining one or more sliding windows comprising a plurality of bits, wherein each bit of the sliding window represents a corresponding packet from the plurality of packets; receiving one or more acknowledgments indicating that the recipient entity has received one or more of the plurality of packets, each of the acknowledgments referencing a respective packet of the plurality of packets; and The value of one or more bits of the plurality of bits in the sliding window corresponding to the one or more received acknowledgments is modified.
18. The computer-readable storage medium of claim 17, wherein: The acknowledgments are received in a second order different from the first order.
19. The computer-readable storage medium of claim 17, wherein: The operations further include: The size of the one or more sliding windows is adjusted by the sender entity based on one or more congestion parameters.
20. The computer-readable storage medium of claim 17, wherein: The plurality of packets include one or more of: a request for a data packet, a data packet, or an acknowledgment.
