Transceiver system with end-to-end reliability and sequencing protocol

Abstract

A packet stream between a transmitter and a receiver in an unreliable and unordered switched packet network can be established as a result of receiving a second packet comprising a second memory operation on a memory address. Transmission of memory store command packets following memory load command packets in the packet stream can be serialized, and a synchronization operation between the transmitter and the receiver can be performed when a packet count at the receiver satisfies a number of data packets in the packet stream.

Description

BACKGROUND

[0001] Communication between processing nodes in a multi-processor system can suffer from gaps in communication error coverage. For example, in a computing system utilizing a signal switching fabric (also referred to herein as a switching network, or simply a 'network') between multiple graphics processing units (GPUs), communication errors occurring in the switching fabric or the GPUs themselves can be unrecoverable and can potentially cause system failure.

[0002] In such a system, when a packet is retransmitted due to a transmission error, the order of packets in the network can change and duplicate packets can be created. A robust protocol would recover not only from transmission errors, but also reorder packets and remove duplicate packets when needed. BRIEF DESCRIPTION OF DRAWINGS

[0003] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0004] Figure 1 An example of a packet-based communication system is depicted.

[0005] Figure 2 An example of packet duplication is depicted.

[0006] Figure 3A An example of in-order packet arrival is depicted.

[0007] Figure 3B An example of out-of-order packet arrival is depicted.

[0008] Figure 4 An example of a serialized load-store sequence is depicted.

[0009] Figure 5A An example of a missing RSP packet for a load operation is depicted.

[0010] Figure 5B An example of a missing RSP packet for a store operation is depicted.

[0011] Figure 5C An example of a missing RSP packet for a store operation following a load operation is depicted.

[0012] Figure 6A An example of a missing REQ packet for a load operation is depicted.

[0013] Figure 6B An example of a missing REQ packet for a store operation is depicted.

[0014] Figure 6CAn example of a lost REQ packet for a store operation after a load operation is depicted.

[0015] Figure 7 An example of a reset timeout when sending a REQ packet is depicted.

[0016] Figure 8 An example of a reset timeout when receiving a RSP packet is depicted.

[0017] Figure 9A An example of a no history filter entry for a load only sequence is depicted.

[0018] Figure 9B An example of a no history filter entry for a store only sequence is depicted.

[0019] Figure 10A An example of a no flow state for a load only sequence is depicted.

[0020] Figure 10B An example of a no flow state for a store only sequence is depicted.

[0021] Figure 11 An example of flow reassembly is depicted.

[0022] Figure 12 An example of exactly once delivery (EOD) for a non-extraction operation is depicted.

[0023] Figure 13 An example of exactly once delivery (EOD) for an extraction operation is depicted.

[0024] Figure 14A An example of a lost REQ packet for a non-extraction operation is depicted.

[0025] Figure 14B An example of a lost REQ packet for an extraction operation is depicted.

[0026] Figure 15A An example of a lost RSP packet for a non-extraction operation is described.

[0027] Figure 15B An example of a lost RSP packet for an extraction operation is depicted.

[0028] Figure 16 An example of exactly once delivery when no flow is available is depicted.

[0029] Figure 17A An example of freeing replay buffers by limiting outstanding REQ packets is depicted.

[0030] Figure 17B An example of freeing replay buffers using a sliding window is depicted.

[0031] Figure 18 A replay buffer management is depicted in accordance with one embodiment.

[0032] Figure 19 An example of network generated duplicate packets is depicted.

[0033] Figure 20 An example of a synchronized transfer is depicted.

[0034] Figure 21 An example of a synchronized transfer using a counter for tracking packets is depicted.

[0035] Figure 22 An example of a lost REQ packet is depicted.

[0036] Figure 23A An example of a lost RSP packet is depicted.

[0037] Figure 23B An example of a duplicate synchronization operation is depicted.

[0038] Figure 23C An example of a lost RSP packet without retransmission is depicted.

[0039] Figure 24 An example of no stream available is depicted.

[0040] Figure 25 An example of no replay buffer available is depicted.

[0041] Figure 26 An example of no reorder buffer available is depicted.

[0042] Figure 27 Embodiments of fast path processing to reduce packet retransmission probability are shown.

[0043] Figure 28 A parallel processing unit 2820 in accordance with one embodiment is depicted.

[0044] Figure 29 A general processing cluster 2900 in accordance with one embodiment is depicted.

[0045] Figure 30 A memory partition unit 3000 in accordance with one embodiment is depicted.

[0046] Figure 31 A streaming multiprocessor 3100 in accordance with one embodiment is depicted.

[0047] Figure 32 A processing system 3200 in accordance with one embodiment is depicted.

[0048] Figure 33 An exemplary processing system 3300 according to another embodiment is depicted.

[0049] Figure 34 A graphics processing pipeline 3400 according to one embodiment is depicted.

[0050] Figure 35 A data center 3500 according to one embodiment is depicted. DETAILED DESCRIPTION

[0051] Described herein are embodiments of communication protocols that provide end-to-end reliability, the logic used to provide one or both of in-order delivery and exactly-once delivery over an unreliable and out-of-order network.

[0052] In certain multi-processor systems, the individual processing nodes communicate over a switched network fabric, including communication of inter-node memory accesses (load and store instructions). Memory accesses can be characterized as in-order or out-of-order, and individual operations are executed at least once or exactly once. Some processing nodes can implement a relaxed memory ordering model. By extending the ordering relaxation to network communication, data transfers benefit from possible performance improvements through techniques such as adaptive routing and multipath- ing, without using potentially expensive reordering logic.

[0053] However, some performance-critical communication patterns can still require ordering. For example, some processing nodes can require sequential consistency for each memory location, which authorizes in-order delivery of accesses to the same memory address. Another example is a bulk data transfer that follows a synchronization operation, where the synchronization operation must be performed before the data transfer completes. Existing reliability and ordering protocols such as TCP or IBRC are not optimized for small, low-latency memory operations (e.g., reads and writes). These protocols also do not natively support network features such as multipath and dynamic adaptive routing.

[0054] Conventional ordering protocols operate like a stream of multiple packets to the same destination address as soon as the first packet of the stream is sent. In effect, the same address stream of packets becomes determined after the first request packet is sent, or more specifically, when a subsequent packet accesses the same destination location.

[0055] Embodiments of the protocol described herein utilize a technique referred to herein as "late-in-flight connection establishment." The disclosed protocol establishes a connection on-the-fly before data is exchanged, without requiring an explicit handshake. The new protocol also defers establishing a connection until a packet that points to the same address as the first packet of a stream and is ordered after the first packet of the stream arrives at the destination.

[0056] Because some switching networks reorder packets, a singleton can be mistaken for the first packet of a stream. The disclosed protocol embodiments utilize an identifier (e.g., a transaction id of the first packet) as a stream id appended to subsequent packets to continue to create the correct association of packets to stream at the receiver.

[0057] Conventional protocols also do not distinguish between the type of operation that constitutes a same address stream of packets. Thus, these protocols do not account for the possible need for a replay buffer for overlapping same address operations to extract data. In some processing systems, an overlapping memory load (LD) operation followed by a memory store (ST) operation requires a replay buffer for operands extracted by the LD in the case that the LD-ST sequence must be retried. This provides the correct observation order. To avoid the utilization of a replay buffer, the disclosed protocol can serialize the LD followed by the ST.

[0058] When a failed LD is reissued, the subsequent LDs (which can have been successfully executed) must also be re-executed to guarantee the correct observation order. The disclosed protocol embodiments can accomplish this by forwarding the (most recently received) LD response to the source memory in order.

[0059] Protocols can implement 'counted writes' to determine when a data packet has been delivered (at the destination) so that a synchronization operation can be performed. These protocols can go back to the source side synchronization when a transmission error occurs. When an error occurs, the protocol can resynchronize the source and destination so that synchronization can be performed on the destination side rather than the source side.

[0060] The disclosed protocol assumes an unreliable and out-of-order network and can include:

[0061] • Late-in-flight stream establishment: A same address stream of packets need not be started when the first packet is sent. In fact, a same address stream of packets is determined after the first request packet is sent, i.e., when a subsequent request accesses the same location. Late-in-flight stream establishment can be utilized so that a stream is established without an explicit handshake prior to switching data. Stream establishment is deferred until the ordered packet after the first packet of the stream arrives.

[0062] • Singleton delineation: Because a network can reorder packets, a singleton can be mistaken for the first packet of a subsequent stream. The transaction id of the first packet can be carried as a stream id appended to subsequent packets to create the association of packets to stream at the receiver.

[0063] • No replay buffer for in-order flow: In case the LD-ST sequence has to be retried, overlapping with a load (LD) followed by a store (ST), a replay buffer for the operand fetched by the LD is needed. This provides the correct observation order. To avoid the need for a replay buffer, the LD can be serialized, followed by the ST.

[0064] • In-order response delivery: When a failed LD is reissued, the following LDs (which can have been executed successfully) can also be re-executed to provide the correct observation order. This can be achieved by forwarding the (most recently received) LD responses to the source memory in order.

[0065] • Error recovery for counted writes: When an error occurs, the source and destination can resynchronize so that synchronization can still be completed on the destination side.

[0066] The following disclosure can be better understood with reference to the following terminology. Other terms shall be consistent with the ordinary meanings in the art, unless the context otherwise requires.

[0067] A request packet (REQ) transmits an operation from the source to the destination. In response, a response packet (RSP) is sent from the destination to the source. If the REQ contains a fetch operation such as a LD, the RSP carries the fetched memory value. The RSP also serves as an acknowledgement (ACK) or negative acknowledgement (NACK).

[0068] The source sends a finish request packet (FIN) to the destination to mark the end of a flow. Receipt of the FIN causes the destination to deallocate the corresponding flow state. The destination sends a finalization response packet (FINACK) to the source to acknowledge receipt of the FIN and deallocation of the flow state.

[0069] After an error occurs, the source sends a get back request packet (GBK) to the destination to request resynchronization of the flow state held at the source and destination. The destination sends a get back acknowledgement response packet (GBKACK) to the source to acknowledge resynchronization.

[0070] The sequence diagrams refer to the following header fields:

[0071] • OP: Operation type

[0072] • TID: Transaction identifier

[0073] • FID: Flow identifier

[0074] • SEQ: Sequence number (starting at 1)

[0075] • EOD: Exactly once delivery

[0076] • ALOD: At least once delivery

[0077] A transaction refers to a request / response exchange.

[0078] A stream refers to a sequence of overlapping requests / operations that must be executed in order. Opening and closing a stream refers to allocating and deallocating, respectively, state to properly order the packets of the stream.

[0079] A unit refers to a request / operation that is not part of a multi-packet stream.

[0080] A packet has a maximum lifetime (or time-to-live (TTL)). That is, a packet does not appear from the network after it expires.

[0081] The example embodiments herein are presented herein with respect to streams comprising memory access commands that require in-order execution to preserve an observation order. However, the techniques are more generally applicable whenever dependencies in a stream of packets require in-order delivery. The in-order delivery of packets carrying memory operations that access the same memory location is merely one example of a data dependency that requires ordering. Other examples of possible applications include synchronization operations and IO traffic.

[0082] Consumer / producer communication patterns often require synchronization after data has been exchanged. An example is a shared producer / consumer queue, where a producer writes items into the queue and a consumer subsequently reads items from the queue. In a distributed system, writing and reading a queue item can involve several packet transmissions. Typically, to indicate that an item was written as well as to indicate that an item was read, a synchronization operation is performed, e.g., to increment a counter after an item was produced / written and to decrement the counter after an item was consumed / read. To avoid errors when accessing the queue, it must be guaranteed that the synchronization operation is only performed after an item was written or read. Thus, the synchronization operation requires in-order delivery in the sense that it must not be delivered before the operation of writing or reading a queue item has been performed. If writing or reading a queue item requires several packet transmissions, there can be no ordering requirement on those transmissions.

[0083] Many input / output (IO) standards, such as Peripheral Component Interconnect (PCI), require ordering. In a distributed system, IO traffic is often transmitted over a network, in which case the network must respect the ordering rules given by the IO traffic. The reasons why IO traffic requires ordering are manifold:

[0084] • By ordering IO transactions, deadlock situations can be avoided.

[0085] • Many IO standards provide backward compatibility with legacy standards that provide ordering.

[0086] • The programming model is simplified by performing IO operations in an order specified by the programmer.

[0087] Thus, the mechanisms disclosed herein can also be applicable to, for example, consumer / producer and networked IO applications.

[0088] Figure 1 is a simplified depiction of the system of a packet-based communication system 100 in one embodiment. Protocol logic is provided in the source switch 126 and the destination switch 130 (with one or more potential intermediate switches 128) that provides for packet-based communication between a source node 102 and a destination node 124. Packet transmission on the access links between the source node 102 and the source switch 126 and the destination switch 130 and the destination node 124 is ordered. The source switch 126 and the destination switch 130 include shared memory 110 and shared memory 112, respectively. The retransmit buffer 104 and the response reordering buffer 120 of the source switch 126 and the replay buffer 118 and the request reordering buffer 122 of the destination switch 130 are implemented in the shared memory 110 and the shared memory 112, respectively. That is, once a packet is received and stored in one of these memories 110, 112, the packet remains in place. "Moving" the packet to one of the reordering buffers 120, 122 or to the retransmit buffer 104 or the replay buffer 118 is reflected in the data structures used to manage the memory and does not require a copy of the packet itself. As a result, no dedicated memory is required to implement the various buffers, and the ordering protocol will typically not exceed the capacity of the buffers once a packet has been logged into the shared memory 110 or the shared memory 112.

[0089] The source switch 126 includes the same address filter 108 that tracks outstanding requests and determines whether a new request accesses an address that has already been accessed by an outstanding request. The destination switch 130 includes a history filter 114 that determines whether an incoming request belongs to a stream for which a first request has already been received. Both the source switch 126 and the destination switch 130 maintain stream state 106 and stream state 116, respectively, to properly order the forwarding of requests and responses.

[0090] In other words, the transmitter determines a first condition whether a subsequent memory access packet to be sent specifies an access to the same memory address as specified by one or more already sent and not completed memory access packets. (A not completed memory access packet is one in which the memory operation specified in the packet has not been acknowledged or otherwise demonstrated to the transmitter as complete.) If this first condition is met, the transmitter tags the subsequent memory access packet with the same identifier as the one or more not completed memory access packets specifying the same memory address. The receiver of the packet determines a second condition whether the received second packet includes the same identifier and a different sequence number as a previously received packet. If this second condition is met, the receiver establishes a packet stream with the transmitter including the previously received packet and the second packet.

[0091] Alternative implementations with protocol logic implemented in the source node 102 and the destination node 124 will be apparent to those of ordinary skill in the art. Such implementations in the nodes can in some cases enhance end-to-end reliability, but can include protocol logic and buffer memory in the nodes resulting in undesirable increases in size, complexity, and power consumption in the nodes.

[0092] Figure 2 Examples of packet duplication are depicted. A sequence of same address (weak) operations requires in-order at least once delivery. For example, delivering the sequence (store 2, store 1) as (store 2, store 1, store 1) over the network 202 is acceptable as depicted in Figure 2

[0093] Figure 3A Examples of in-order packet arrival are depicted, and Figure 3B Examples of out-of-order packet arrival are depicted. These instances describe a typical use case in which the resources at the destination appropriately establish a stream and store the REQs in a reordering buffer as needed. The examples depict a stream including four REQs with store (ST) operations for the same memory address. The REQs carry sequence numbers SEQ.

[0094] In Figure 3A , the REQs arrive in-order, and in Figure 3B , they arrive out-of-order (OOO). The REQs arriving OOO are temporarily stored in a reordering buffer. In the example shown in Figure 3B , REQ2 and REQ4 arrive OOO and are held in the reordering buffer until they become in-order once REQ1 and REQ3, respectively, are received. Once the REQs have been delivered and the RSPs returned, a FIN / FINACK handshake occurs to close the stream at the destination and source.

[0095] ​Unlike conventional connection-oriented protocols where a connection is explicitly established, a connection (also referred to herein as a flow) is opened 'on the fly'. This means that the transmitting end of a communication opens a flow when a REQ is injected into the network if the address filter indicates that there is already an outstanding REQ with an operation for the same remote address. On the receiving side, a flow is opened when the first REQ with SEQ > 1 is received. The REQ with SEQ = 1 can be the first packet of a flow or a single piece and thus the opening of the flow is delayed until a subsequent REQ with SEQ > 1 is received. When a REQ with SEQ = 1 is received, an entry is made in the history filter so that it can be determined later whether the first packet of the flow has been received.

[0096] In Figure 3A REQ1-REQ4 arrive in order at the destination. First, REQ1 is received and forwarded to the destination memory. The receipt of REQ1 is recorded with an entry in the history filter. Next, REQ2 arrives at the destination and opens a flow. The query filter shows that REQ1 has been received. From this, REQ2 is in order and can be forwarded. When REQ3 arrives, the query of the flow state shows that REQ2 is the last in order REQ received, and thus, REQ3 is in order and can be forwarded. REQ4 is handled similarly. The sequence ends with FIN / FINACK, which removes the flow state kept at the source and destination.

[0097] In Figure 3B REQ2 is the first REQ to arrive at the destination and open a flow. From the history filter, REQ1 has not been received. Thus, REQ2 is received out of order and is forwarded to the reordering buffer. REQ1 is then received and forwarded. Each time a REQn is forwarded, the reordering buffer is checked for REQn+1. In this example, REQ2 as well as REQ4 are found in the reordering buffer when REQ1 and REQ3 are received, respectively. Once the REQs in the reordering buffer are in order, they are released and forwarded to the destination memory. Responses can also arrive out of order as shown in the example. While the receiver can reorder the responses (e.g., using a scoreboard that tracks outstanding requests), the ordering protocol can further employ a response reordering buffer that forwards responses in order. This provides the correct observed order when a retransmission occurs.

[0098] In one embodiment, the history filter stores tuples {src_id, fid}, where src_id identifies the sender and fid identifies the stream. The tuples serve as unique identifiers for the same address stream. The tuple is written when a REQ with SEQ = 1 is received; for REQ1, FID = TID. False negatives are possible in the sense that REQ1 is received and forwarded but not found in the history filter. The handling of false negatives is described later. In addition, the inclusion of TID is described later.

[0099] In summary, the packet forwarding rules can be implemented as follows. Request packet REQn is forwarded to the destination memory:

[0100] • if n = 1,

[0101] • if n = 2 and the history filter has an entry for the corresponding tuple,

[0102] • if REQn-1 has already been forwarded according to the stream state.

[0103] Figure 4 Examples of serialized load-store sequences are depicted. If (i.e., the condition) the stream contains only loads, only stores, or contains stores followed by loads, the packet transfer is overlapped. If a load is followed by a store, the load is serialized. In Figure 4 REQ2 includes a load, and REQ3 includes a store operation. REQ3 is held by the source until REQ2 has completed.

[0104] Serializing load followed by store can avoid the need for a replay buffer. If these operations overlap, the destination will maintain the load response in a replay buffer to avoid false results in case the load needs to be retransmitted due to a transmission error after the store is executed. Eliminating the need for a replay buffer for loads is particularly effective when the load / store sequence occurs at the beginning of the stream, and thus in some embodiments, serialization can be implemented particularly for this case. Without serialization, the destination can need to store every load response in a replay buffer, even for single items.

[0105] If the load / store sequence occurs later in the stream, overlapping transmission of the load response in a replay buffer and the store can be practical, as at that time the destination can have the record of the stream.

[0106] Figure 5A Examples of missing RSP packets for load operations are depicted. Figure 5B Examples of missing RSP packets for store operations are depicted. Figure 5C Examples of missing RSP packets for store operations following load operations are depicted. These examples assume that the stream state and the reordering buffer are available and utilized.

[0107] When the operation depicted fails, the source may initiate a "return" handshake that includes a GBK request sent from the source to the destination. The GBK request specifies the SEQ of the failed REQ, followed by a GBKACK response sent from the destination to the source. Upon completion of the GBK / GBKACK handshake, the source retransmits the failed REQ along with subsequent REQs. In the depicted example, three requests, REQ1..3, are sent and forwarded to the destination memory. RSP1 is lost, while RSP2..3 successfully returns to the source. Once REQ1 has timed out, the source sends a GBK (SEQ=1) to inform the destination that the first operation has failed.

[0108] The destination responds slightly differently to the loading and storage of retransmissions. Figure 5A In the load-only sequence shown, the retransmitted loads (REQ1..3) are re-executed. This ensures the correct observation order. If only REQ1 is retransmitted, the execution of the loads will be reordered.

[0109] Figure 5B A similar scenario for storage operations is described. In this case, the operation is only acknowledged by the destination, not re-executed, to guarantee the order of observation. If a storage operation, or a series of storage operations, is re-executed, the destination node might read values ​​in the wrong order. Figure 5B In the example shown, retransmissions of REQ1-3 may be unnecessary. Receiving RSP2 implies that the destination has already received REQ1. Therefore, backtracking and subsequent retransmissions of REQ may not be required. Configuring exception handling in this way is simple and uniform, reducing protocol complexity. Because exceptions are rare, they are tolerable and... Figure 5B Inefficiency in certain situations.

[0110] Figure 5C The sequence of storage used after loading is described. The same rule applies: storage is confirmed and loading is re-executed.

[0111] In summary, the receiver can respond to the echo request as follows:

[0112] Remove all REQs from the reorder buffer.

[0113] • The retransmitted load is forwarded to the destination memory.

[0114] • Retransmitted storage is only forwarded to the destination storage if (under such conditions) they were not previously forwarded.

[0115] The packet destination uses a sequence pointer specifying the next request to be forwarded and also records the last forwarded storage, such as... Figure 5CThe sequence pointer is reset to 1 (one) by the GBK, but the retransmitted REQ1 is not forwarded to the destination memory, because the destination has recorded that REQ1 has already been executed.

[0116] The retransmission can cause duplicate load responses (e.g. in Figure 5A RSP2 and RSP3 are received twice in Figure 5A The RSP2..3 packets received before the GBK are discarded. This means that the load responses should be forwarded to the source memory in order, and if multiple copies of some response packet RSPn are received, the most recently received RSP packet should be forwarded. Otherwise, if out-of-order RSPs are forwarded to the source memory, an outstanding RSP with a lower sequence number can cause a timeout, so that the load needs to be re-executed, invalidating the already forwarded responses. This is the case in Figure 5A for the first time before the return handshake. The in-order delivery of responses requires the response buffer to hold the RSPs until they are in order.

[0117] Alternative (more complex) implementations to handle transmission errors can be considered. In Figure 5A , Figure 5B and Figure 5C RSP1 can be derived from RSP2 when RSP2 is received by the source, and after a timeout due to the loss of RSP1. The reception of RSP2 by the source means that the destination has executed REQ1, even if RSP1 was not received. In Figure 5A the value returned by RSP2 can be used as the return value of (missing) RSP1. Even if the destination memory has changed and the actual fetch values of REQ1 and REQ2 are different, it is acceptable to use the return value produced twice by REQ2, because the observation order is followed. In Figure 5A , Figure 5B and Figure 5C the reception of RSP2 can be interpreted as an ACK of REQ1.

[0118] Figure 5A , Figure 5B and Figure 5C The examples in Figure 5B If no stream is available when the first copy of REQ2 is received in

[0119] Figure 6A depicts an example of a missing REQ packet for a load operation. Figure 6BExamples of lost REQ packets for store operations are described. Figure 6C Examples of lost REQ packets for store operations after a load operation are described.

[0120] These examples involve the loss of REQ packets. In the example, REQ1 is lost, so the destination receives REQ2..3 out of order and stores these packets in the reordering buffer. Since RSP1 is lost, the timeout timer eventually expires, causing the return handshake. This removes REQ2 and REQ3 from the reordering buffer and sets the sequence pointer to 1 (one). In all three cases, REQ1..3 are resent and eventually executed, with no operations performed before the return handshake. At the time of receiving the GBK, the stream at the receiver is already in place. It does not have to be the case. Thus, when the GBK is received and no receiver stream is open, a new stream can be opened.

[0121] Timeout timer

[0122] Figure 7 Examples of resetting the timeout when sending a REQ packet are depicted. Since packets have a maximum lifetime, a timeout timer can be used to detect the loss of a packet. A timeout timer can be started for each packet sent into the network. However, checking the timer for each outstanding packet creates a significant overhead. The scheme described below uses only one (or two) timers per stream.

[0123] A single timer per stream can be reset each time a REQ is sent. If the timer expires and there are outstanding REQs, then a REQ or RSP has been lost, and the GBK / GBKACK sequence is triggered. In the example, the timer is started when REQ1 is sent, and the timer is reset when REQ2 and REQ3 are sent, respectively. Since RSP1 is lost, the timer times out, triggering the GBK / GBKACK handshake. Figure 7

[0124] When a GBK or FIN is sent to catch errors that occur during the GBK / GBKACK and FIN / FINACK handshakes, the timeout timer is also started / reset. If any of these packets are lost, a timeout will occur and the corresponding handshake is retried.

[0125] The timeout time must be set to a value that is greater than twice the maximum packet lifetime (to cover REQ as well as RSP forwarding time). When the timeout timer expires, it has high confidence that no more RSPs will be received.

[0126] ​For long flows, it can potentially take a long time to detect an error, as the timer is reset at every send of a REQ. In addition, a large amount of reordering buffer space can be consumed, as every REQ received after a lost REQ is considered out of order. Different measures can be implemented to prevent this. A maximum flow length can be enforced, which limits the time taken to report an error and also limits the number of entries in the retransmission and reordering buffers occupied by the flow. Another option is to add a mechanism at the destination to detect no progress. Such a mechanism can use a timer to detect that an entry has not been forwarded for a certain configured period of time, or limit / restrict the flow's consumption of the reordering buffer. No progress can be reported back to the source through a NACK, which indicates a timeout or excess of the used buffer, respectively.

[0127] Figure 8 An example of resetting the timeout when receiving a RSP packet is depicted. The timeout timer is reset when the next in-order RSP is forwarded to the source memory (instead of when a REQ is sent). This scheme relies on the in-order return of RSPs to the source memory. As this is already implemented in the ordering protocol, no additional logic is needed. An additional timer is used to ensure that an outstanding REQ has timed out before the return handshake starts. This timer is reset at every send of a REQ.

[0128] In Figure 8 timer 1 is the timer that is reset when the next in-order RSP is received. Timer 2 is the timer that is reset every time a REQ is sent. In this example, RSP2 is lost, as captured by the timeout of timer 1, which is reset upon reception of RSP1. The protocol waits for timer 2 to expire to ensure that all RSPs are received before the return handshake. RSP3 arrives while waiting for timer 2 to expire. In Figure 8 timer 2 is depicted as being reset when REQ3 is sent.

[0129] The advantage of this scheme is that the time to detect packet loss is bounded by 2*max_packet_lifetime, and no restrictions on the flow length need to be imposed.

[0130] Figure 9A An example of a history filter entry that only loads the sequence is depicted. Figure 9B An example of a history filter entry that only stores the sequence is depicted. These cases are receivers that lack some of the resources needed to implement line rate overlapping transmissions.

[0131] In the case that the history filter is implemented as a cache, the entry can have been evicted when a subsequent REQ queries the filter. Figure 9A and Figure 9BREQ2 and REQ3 arrive at the destination and no entry for REQ1 is found in the history filter. Thus, REQ2 and REQ3 are considered out-of-order and stored in the reordering buffer. Since no RSP is generated, REQ2..3 will eventually time out and cause a round-trip handshake, followed by retransmission of REQ2..3. This process is identical to the process for handling transmission errors.

[0132] Figure 10A Examples of flowless state for load-only sequences are depicted. Figure 10B Examples of flowless state for store-only sequences are depicted. In these examples, when REQ2 arrives, no flow is available. When REQ3 arrives, a flow is available and REQ3 is stored in the reordering buffer. As a result, RSP2 is returned with a NACK. The NACK is handled like a transmission error, eventually causing a timeout that triggers a return handshake. In one embodiment, the destination can discard REQ2 and not return RSP2 at all.

[0133] Figure 11 Examples of flow reassembly are depicted. Because REQs can be reordered in the network, the arrival of a single can be interleaved with the arrival of REQs of a subsequent flow. If all REQs carry operations that access the same address, additional information is needed to disambiguate the single and the flow. Figure 11 In this example, the single (highest REQ and RSP) is followed by a flow (remaining packet transmission) with all REQs carrying operations for the same destination address. In this example, REQ1 and REQ2 of the flow (not the single) arrive out-of-order. When REQ2 arrives first, it should not be considered in order by mistakenly inferring that the previous single was the first REQ of the flow.

[0134] A mechanism can be used to distinguish between a single and a subsequent flow. In one embodiment, a 'late binding approach' is used to associate a REQ with a flow by reusing the TID of the first request as the FID for the subsequent REQs of the flow. In Figure 11 In this example, when REQ1 of the flow is sent, it is not known whether the flow is started, therefore, no FID is provided. The transmission of REQ2 establishes the flow. The FID is attached to REQ2 and the subsequent REQs. In order to be able to associate the REQ1 of the flow with the following REQ2-3, the FID is set to the value of the TID of the REQ1 of the flow. This way, the destination has enough information to correctly reassemble the flow (and depict it from the preceding single).

[0135] The destination can use a history filter to determine whether REQ2 is in order; that is, whether REQ1 has already been received and forwarded. In Figure 11In the example of Figure 1, REQ2 arrives at the destination first, at which time no flow state has been assigned. The history filter is looked up using the key <source_id, FID=2>. No entry is found and REQ2 is correctly considered out-of-order.

[0136] To help ensure correct operation, entries in the history filter should expire before the corresponding TID is recycled and reused. To time out entries in the history filter, they can be provided with a timestamp when they are entered. When they are retrieved and the timestamp indicates that they have expired, the entries can be ignored.

[0137] In a pool of n TIDs with at most m outstanding REQs, there are n-m 'free' TIDs. If the free TIDs are stored in a FIFO and recycled, at least (n-m)*t_s time is spent before a TID is reused, where t_s is the packet serialization time. For example, if m=1000, n=5000 and t_s=32*8 bits*1 / 200 Gbit / s=1.28 ns (where 32 bytes is the minimum packet size and 200 Gbit / s is the transmission rate), the reuse interval >=4000*1.28 ns=5.2 us, which in turn determines the maximum lifetime t_h of an entry in the history filter.

[0138] The size of the history filter is determined by the amount of time a REQ1 packet must be remembered. In Figure 11 In the example of Figure 1, REQ2 arrives at the destination first, at which time no flow state has been assigned. The history filter is looked up using the key <source_id, FID=2>. No entry is found and REQ2 is correctly considered out-of-order.

[0139] Another constraint as mentioned above is the maximum allowed lifetime t_h of an entry in the history filter. If t_h=5.2 us as in the example above, taking into account the two constraints, e*t_s<min(RTT, t_h) results in e=4063 entries.

[0140] If t_h<RTT, the source must guarantee that REQ2 is injected at most t_h time after REQ1 is injected. If this timing constraint cannot be met, the injection of REQ2 must be delayed until after RSP1 is received (in which case REQ2 becomes another REQ1).

[0141] Non-idempotent operations, such as atomics, can require exactly-once delivery. To be able to detect duplicates, a stream can be utilized to track requests that have already been delivered. As opposed to the protocol described above for at-least-once delivery (ALOD), exactly-once delivery (EOD) can require a stream to be established as soon as the first request of the stream is received.

[0142] A 'duplicate' is a packet that is a copy of a previous packet, and thus indistinguishable from the original (previous) packet. The duplicate and original packets do not necessarily have to exist in the network at the same time.

[0143] In addition to the stream state, EOD can further require a replay buffer if the operation extracts data in order to be able to'replay' the responses of a failed REQ / RSP transaction. The protocol for EOD can use the timeout and return mechanisms described previously to handle transmission errors.

[0144] Figure 12 An example of exactly-once delivery (EOD) for non-extraction operations is depicted. Figure 12 The example in FIG. 6 shows a sequence of three non-extraction reduction operations (for the same address). This sequence diagram is similar to some of the sequence diagrams discussed previously, except that a stream is opened as soon as the first REQ arrives at the destination. Because non-extraction atomics do not return a value, a replay buffer is not required.

[0145] Figure 13 An example of exactly-once delivery (EOD) for extraction operations is depicted. Fetch operations, such as atomics, can require the extracted data to be stored in a replay buffer until it is confirmed that the source has received the data. The extracted value is again returned to the source memory in order. Although EOD does not necessarily do this because there cannot be any duplicate RSPs, the returned values can be ordered to homogenize the protocol. Mechanisms for releasing the replay buffer are discussed below. Although these examples depict flows that include the same operation type (extraction or non-extraction), the flows can more generally include a mix of these operation types.

[0146] Figure 14A An example of a REQ packet loss for non-extraction operations is depicted. Figure 14B An example of a REQ packet loss for extraction operations is described. Figure 15A An example of a RSP packet loss for non-extraction operations is described. Figure 15B An example of a RSP packet loss for extraction operations is described.

[0147] The same timeout mechanisms and return handshake described previously can be used to detect transmission errors and resynchronize the source and destination, respectively, before a transaction sequence that retries in a failed REQ / RSP transaction.

[0148] Figure 16 An example of exactly once delivery when no stream is available at the destination is depicted. No stream is available at the destination when REQ1 arrives. Therefore, REQ1 is discarded and negated. When REQ2 arrives, a stream becomes available and REQ2 is put in the reordering buffer. NACKs are handled similarly to transmission errors as described above. A return mechanism can be used to homogenize the protocol. When an EOD REQ / RSP transaction fails, it can be retransmitted without resetting the stream state.

[0149] Figure 17A An example of freeing the replay buffer by limiting outstanding REQ packets is depicted. Figure 17B An example of freeing the replay buffer using a sliding window is depicted.

[0150] The reordering buffer can be freed as soon as the REQ packet has been forwarded to the destination memory. The replay buffer cannot be freed until the corresponding response packet is successfully delivered to the source. The FIN / FINACK handshake terminating the stream eventually frees all replay buffers. Additional mechanisms can be exploited to limit the number of replay buffers used by a stream. This can be particularly advantageous for long sequences of accesses to the same address, where the process can be hindered when too many replay buffers are occupied.

[0151] Figure 17A A solution is described in which the source limits the number of outstanding REQs. Assume m is the maximum number of outstanding REQs, the reception of REQ(SEQ=n) tells the receiver that the REQs with SEQ <= n-m in the replay buffer can be freed. In this example, m=2. Thus, the reception of REQ3 removes RSP1 from the replay buffer.

[0152] Another solution, as depicted in Figure 17B is the addition of a sliding window mechanism using the "ack sequence number" field ASEQ that makes the source communicate to the destination what is the highest received in-order sequence number. The destination can use this information to remove from the replay buffer any RSPs of this stream with SEQ <= ASEQ. In Figure 17B REQ3 specifies ASEQ=1 to make the destination know that RSP1 has been received and the corresponding replay buffer can be freed.

[0153] A robust protocol should handle the case of resource exhaustion. A robust protocol should work normally when resources are exhausted and actions are taken to prevent resource starvation.

[0154] Resources that can be limited include:

[0155] • Source-side stream: the source stops transmitting REQs until the stream is freed.

[0156] • Re-transmission buffer: Once a source switch accepts a REQ, space in the re- transmission buffer is guaranteed. See note 1 below.

[0157] • Response buffer:

[0158] o Option 1: Allocate response buffer before issuing REQ with fetch operation. This applies to ALOD and EOD. See note 2 below.

[0159] o Option 2: When no response buffer is available, the source drops RSP, causing re- transmission of the corresponding REQ and replay of RSP.

[0160] • Destination side flow: If no flow is available, the destination NACKs the REQ. See note 3 below.

[0161] • Re-ordering buffer: Once a destination switch accepts a REQ, space in the re-ordering buffer is guaranteed. See note 1 below.

[0162] • Replay buffer: Allocate replay buffer before a fetch operation is forwarded to the destination memory (EOD only). See note 2 below.

[0163] Note 1: The central memory switch can guarantee that once a packet is accepted, it can be placed into either the re-transmission buffer (source switch) or the re-ordering buffer (destination switch). If no space is available in the memory, the packet can be dropped, so that the protocol handles this case as a transmission error.

[0164] Note 2: Buffer space for RSP needs to be reserved before forwarding the corresponding REQ and performing the transmission operation. If no space is available in the response buffer in the source switch (option 1), the sending of the corresponding REQ can be postponed until space becomes available. However, this technique does not work if no space is available in the replay buffer in the destination switch. Because packets have a maximum lifetime, the transmission cannot be delayed for a significant amount of time. From this, if no space is reserved in the replay buffer, the REQ ready to be forwarded to the destination memory needs to be negated. To prevent starvation, the destination should reserve replay buffer space for at least one RSP. That is, the destination must open a flow only if space in the replay buffer is available. Similarly, when the destination opens a flow to guarantee progress, space for at least one REQ in the central memory should be reserved.

[0165] Note 3: Starvation can occur if a source's REQ is repeatedly negated when the destination side flow state has been exhausted. Known techniques such as exponential backoff or resource reservation protocols can be used to mitigate starvation.

[0166] On the source side, the maximum number of flows is at most the maximum number of outstanding requests. To estimate the number of flows required at the destination, assume that there are n sources, each injecting packets for one flow, and all flows go to the same destination. Further assume that there are no packet collisions in the network, and that the total injection rate of all flows is equal to the total injection rate of all flows. In the worst case, the n sources inject the same address request maximally spaced by the round trip time (RTT). From this, there can be at most n = RTT / t_s active flows at the destination. For example, in the case of RTT = 5us and t_s = 1ns, the maximum number of 5k flows is required.

[0167] Figure 18 Replay buffer management is depicted according to one embodiment. Once an extracted non-idempotent operation is forwarded to the destination memory for its execution, it must be guaranteed that the replay buffer has space for the extracted value. The space for the extracted value must be allocated in the replay buffer before the REQ is forwarded to the destination memory. Figure 18 Two instances of REQs that are denied because there is no space in the replay buffer are shown. Assuming that the packet has a maximum lifetime, it is not an option to keep the REQ until space is available in the replay buffer.

[0168] When replay buffer space becomes scarce, the flow can starve. To prevent this, a technique such as allocating replay buffer space to the source that requests to open a flow in a round-robin fashion can be applied. A single replay buffer is sufficient to make the flow progress. Thus, the right strategy can be for the destination to open a flow for extracting EOD operations only if replay buffer space is available. To guarantee progress, at least one replay buffer can be reserved when the flow is opened.

[0169] Certain types of switches, such as Ethernet switches, can create duplicate packets. For example, when some switches receive a frame and do not find its destination address in the MAC table, they flood all the output ports.

[0170] To distinguish between the duplicates generated in the network and the duplicates generated by the source (caused by retransmission), a version field can be added in the packet header. The receiver can use a de-duplication filter indexed by the key <source_id, TID, version> that guarantees at most one delivery. The entry must stay in the de-duplication filter as long as the network-generated duplicates can appear at the receiver or the packet has not expired (if that time is shorter than the previous one).

[0171] The duplicate removal of REQs of SEQ>2 does not require additional mechanisms. Therefore, the version field is only needed for REQs with SEQ=1 (which can be the first REQ of a piece or stream). If very few retransmissions are expected, Huffman coding can be used for the version field, adding only one bit to the packet that is not retransmitted. The same mechanism can be used for REQ, GBK, and FIN packets.

[0172] Conventional mechanisms handle duplicates on the return path. For example, some graphics processing units (GPUs) maintain a scoreboard indexed by TID and use it to de-duplicate responses. The scoreboard must remember the TID as long as a copy can reach, and in addition, the TID is not reused for that amount of time. Other return packets, such as GBK ACK and FIN ACK, can be handled in the same way as conventional RSPs.

[0173] Figure 19 An example of network-generated duplicate packets is depicted. Duplicate REQ1 (VER=1) is removed by the destination-side duplicate filter, and duplicate RSP1 is removed by the (e.g., GPU) scoreboard.

[0174] Figure 20 An example of synchronous transfer is depicted. Synchronous transfer (STR) includes the out-of-order transfer of data packets following a synchronization operation. STR enables target-side synchronization, thereby avoiding any RTT delay between the end of data transfer and the synchronization operation. In Figure 20 In the example, there are three data packets REQ1..3 followed by a synchronization operation in REQ4. REQ1..3 are forwarded to the destination memory in the order they arrive. REQ4 includes a synchronization operation that requires EOD and is only forwarded to the destination memory after all data packets have been delivered. In this example, REQ4 arrives at the destination before all packets are forwarded to the destination memory, and is therefore temporarily held in a reordering buffer until the data portion has completed.

[0175] Some bookkeeping is used to determine when the synchronization operation can be released. One scheme that can be utilized uses a count of writes. A counter tracks the data packets at the destination to determine when the synchronization operation can be executed. When the counter has accounted for all data packets, the condition for executing the synchronization operation is considered to be met.

[0176] Figure 21An example of a synchronous transfer using a counter for tracking the packets is depicted. The STR comprises three data packets REQ1..3 for a store operation, followed by a sync packet REQ4 for a sync operation. REQ4 carries a count CNT=4 which specifies the total number of packets that make up the STR. The destination counts the REQ packets and when the counter reaches CNT, the condition for forwarding and performing the sync operation is met. In this example, REQ1 and REQ3 arrive out of order. The order of the data packets REQ1..3 does not matter, so the destination forwards them to the destination memory as they arrive. However, REQ4 with the sync operation must be ordered according to the packets. Since it arrives out of order, it must be stored in a reordering buffer until after the last data packet REQ3 is received.

[0177] The data and sync packets are associated with a flow identifier FID which is used by both source and destination to track the transfer. The flow state required at the destination is the aforementioned packet counter. The destination opens the flow upon receiving the first REQ, which in this example is REQ2. The flow state at source and destination is removed by a FIN / FINACK handshake. In Figure 21 In the example, it is assumed that resources such as flow state and reordering buffer are available and not exhausted.

[0178] A robust STR protocol should include mechanisms to recover from transmission errors. Simply re-sending the REQs in response to a transmission error can not be sufficient, because the destination can count the REQs multiple times (if the RSPs are lost), loosing synchronization. To avoid this, the 'back' handshake introduced above can be used to reset the destination counter and re-synchronize the source and destination state.

[0179] Figure 22 An example of a lost REQ packet is depicted. A timeout timer is used to detect the loss of a packet. Before retransmission, a GBK / GBKACK exchange takes place which informs the destination to reset the packet counter to 0 and to remove the sync operation from the reordering buffer if it has been received.

[0180] The timeout timer can be implemented in a number of ways, for example:

[0181] 1. Each REQ is associated with a timeout timer. The timer is started when the REQ is sent and stopped when the corresponding RSP is received. When the timer times out, an error occurs. When it is guaranteed that no more RSPs will be received, no more REQs are sent and another timer for the expiration is started.

[0182] 2. Use a single timeout timer, which is reset every time a REQ is sent. When all RSPs have been received, the timer is stopped. When the timer times out, one or more REQ / RSP transactions fail.

[0183] The first option has the advantage that transmission errors are detected faster than in the second option, where the error would only be detected when the timeout timer expires, which is reset when the last REQ is sent. On the other hand, the first option is more complex, as it requires a timeout timer for each REQ, while the second option uses only one timer. Some of the following figures depict the use of the second option. The timeout timer can also be used to catch errors that occur during the GBK / GBK ACK and FIN / FIN ACK handshakes.

[0184] After the GBK / GBK ACK handshake, transmission resumes. That is, the timed-out packets and the packets that have not been transmitted are sent. In this example, REQ 1 is re-sent, as it was lost on the first attempt, and REQ 4 is re-sent with the sync operation, as it was not acknowledged. The CNT count in the re-sent REQ 4 has been adjusted to 2 to reflect the number of newly sent REQs.

[0185] Figure 23A to Figure 23C A scenario is depicted in which an RSP is lost. In Figure 23A RSP 3 is lost. The timeout timer expires and triggers the'return' handshake. After the GBK / GBK ACK completes, REQ 3 is re-sent and the remaining REQ 4 is sent. The re-sent REQ 4 specifies CNT = 2, reflecting the number of REQs sent after the'return' handshake.

[0186] In Figure 23B RSP 3 is lost, as well as RSP 4. Although the STR completes, the source is not aware of this fact and re-sends REQ 3 and REQ 4 after the'return' handshake. Because the memory cannot be modified after the sync, the destination cannot forward REQ 3 and REQ 4. This example shows that deduplication is required not only for the sync operation, but also for any data packets that arrive after the sync.

[0187] Figure 23C Another scenario is shown in which an RSP is lost. Here, the data transfer REQ 3 / RSP 3 fails. At the time the timeout timer expires, RSP 4 arrives, acknowledging that the sync operation is performed. Because the receipt of RSP 4 indicates that the STR has completed, no retransmission occurs.

[0188] Figure 24An example is depicted where no flow is available. As mentioned above, a flow is opened when the first packet of the flow arrives at the destination. However, if the destination has run out of resources, it is not possible to open the flow. The resources needed to open a flow include a counter to keep track of the number of REQs received, and a replay buffer if the synchronization operation is a fetch value. In Figure 24 REQ1 is the first packet to arrive at the destination. At this point, the flow cannot be opened. Since REQ1 is a data packet, it can still be forwarded to the destination memory. When REQ2 arrives next, the flow is opened. However, REQ1 is not counted, and when REQ4 with a synchronization operation arrives, as a result, it is stored in the reordering buffer. REQ4 will eventually time out and cause a "return" handshake. REQ4 is resent, now CNT = 1, and forwarded to the destination memory.

[0189] If a replay buffer is needed, it can be allocated either when the flow is opened or when a synchronization operation arrives at the destination. Considering the former approach, if no replay buffer is available, the flow is not opened. And regarding the latter solution, if no replay buffer is available to store the fetched value, the REQ with a synchronization operation is negated and resent until a replay buffer becomes available. Figure 25 The latter solution is depicted in where a scenario where no replay buffer is available. REQ4 with a synchronization operation is retried until a replay buffer becomes available.

[0190] Another resource that can run out is the reordering buffer. Figure 26 An example is depicted in where a reordering buffer is needed when REQ4 with a synchronization operation arrives at the destination before all data packets are received. If no reordering buffer is available, the REQ is negated. This will cause REQ4 to time out and be retransmitted. This situation is similar to the previous case where no replay buffer was available. When the source receives a NACK, it does not know which resource is missing. The difference is that no reordering buffer is needed to make progress, while a replay buffer is needed if the synchronization operation fetches a value.

[0191] Computing devices, such as central processing units and graphics processing units, implement a memory consistency model that provides guarantees about the order in which a set of events appear to have occurred. The memory consistency model is conventionally a problem for memory systems to implement because it requires ordering of memory operations (loads and stores) within a chip. Some memory systems, such as Nvidia's NVLink and NVSwitch networking systems, extend the memory model across many computing devices by transmitting memory operations over a packet-switched network. As the network system scales from tens of endpoints to thousands of endpoints, performing packet ordering in the network becomes burdensome and degrades performance. If the network cannot guarantee packet ordering, it will violate the memory consistency model. Because enforcing ordering in the network degrades performance, the endpoint devices (the ultimate destination of the transmitted memory operations) can reorder the packets to enforce the memory consistency model. To reorder the packets, the endpoint places the packets into a reorder buffer until all other related packets arrive, and the endpoint properly reorders the packets according to the memory consistency model.

[0192] A history filter can be used at each endpoint to identify recently received packets. When the first packet in a stream arrives at an endpoint, the endpoint allocates an entry in the history filter. At this point in time, the endpoint does not know whether the packet is a singleton or the first packet in a longer stream. If the packet is a singleton, it will be evicted from the history filter after a timeout period elapses. If the packet is the first packet in a plurality of packets in a longer stream, then subsequent (first) packets in the stream will eventually arrive at the endpoint, and in response, the endpoint will establish the stream by looking up the first packet of the stream in the history filter. However, if the delay between the first packet and the second packet is longer than the eviction period of the history filter, this will not happen.

[0193] The history filter uniquely identifies a stream based on a source identifier (e.g., source endpoint) and a stream identifier (e.g., memory address). The history filter can be implemented as a small cache of recently accessed arrived packets. A packet is kept in the history filter for a preset duration, and then evicted. If a second packet of a stream (i.e., packet B) arrives and the first packet (packet A) is evicted, packet B exists without the context of whether packet A arrived. In this case, packet B will be held in the history filter waiting for the arrival of packet A, which has already arrived and been evicted from the history filter. No stream is established. This eventually triggers an error recovery mechanism that determines the correct order of events but wastes bandwidth and time.

[0194] A "fast path" mechanism can be used to reduce the probability of the history filter evicting an entry when a subsequent packet is still in the network. This prevents the opportunity for evictions from the history filter to result in undesirable retransmissions.

[0195] As explained above, large skew between packets within the same flow can cause unnecessary retransmissions when the historical filter evicts the flow before other packets of the flow arrive. To reduce the skew between packets that need to be ordered, a mechanism can be implemented to "fast path" the second packet in a flow. The second packet establishes the flow at the destination, where the flow is evicted after a predetermined amount of time. With the "fast path" packet, it will traverse the network quickly, thus reducing the likelihood of packets arriving after the first packet in the flow is evicted by the historical filter.

[0196] By limiting the delay of the packet that establishes the flow, the system can evict or move the individual to a reordering buffer from the historical filter more quickly. In conventional approaches, limiting the delay of the flow establishing packet can involve limiting the amount of traffic in the network. Much of the delay in the network can be sourced from packet queuing delay, and limiting the traffic limits the amount of queuing delay, as the size of the queue is limited by the number of packets in flight. Limiting the traffic negatively impacts the throughput of the network. By limiting the number of packets that each end host can inject, the end hosts cannot saturate the network, which reduces the overall throughput of the network. This means that for some applications, reducing the network throughput to reduce packet delay is not an option.

[0197] In one embodiment, a mechanism is utilized to reduce and / or limit the delay of the packet with sequence number two (2), which enables the network to maintain high throughput and reduce / limit the size of the buffer. The second packet in the sequence establishes the flow. If the second packet is guaranteed to arrive within some bound, then the individual can be quickly evicted from the historical filter, as the second packet in the flow arrives, or the first packet is identified as an individual rather than part of a longer flow.

[0198] In one embodiment, each sender is limited in the number of outstanding packets that they can have on the "fast path." Algorithms are known in the art for determining the number of outstanding packets that each sender can have and maintain a delay on packets bound to: see, e.g., GROSVENOR, M. P., SCHWARZKOPF, M., GOG, I., WATSON, R. N., MOORE, A. W., HAND, S., AND CROWCROFT, J. Queues don't matter when you can{JUMP} them! In 12th Symposium on Networked Systems Design and Implementation ({NSDI} 15) (2015), pp. 1-14. The bound on the number of outstanding packets directly relates to the size of the historical filter resources. If the bound is lowered, then fewer resources need to be allocated to the historical filter.

[0199] In one embodiment, a "fast path" is implemented using virtual channels. Virtual channels enable separate and more granular queuing control for different types of network traffic. Each packet can be assigned a virtual channel, and the packet consumes the resources of its assigned virtual channel. Switches can use bandwidth allocation policies to arbitrate between virtual channels. In one embodiment, the fast path utilizes strict priority queuing and allocates one virtual channel that preempts traffic on any other virtual channel. This enables packets on the fast path virtual channel to skip packet queues and arrive at the destination with lower latency. The fast-arriving packets enable the destination to quickly evict single pieces from the history filter, which reduces the required history filter size.

[0200] As described above, access to the "fast path" can be limited, so the fast path does not become congested. Some embodiments can utilize multiple "fast paths" with different priorities, each of which can preempt lower-priority fast paths in case of congestion / conflict.

[0201] In some embodiments, the fast path can be implemented using age-based arbitration. Age-based arbitration classifies packets by their age. The older the packet, the higher its priority. By setting the age of the setup flow packet to a very large value, or to infinity, the packet can move quickly through the network before other packets with lower ages. By limiting the number of outstanding packets that each sender can send with high age, the delay of the setup flow packet is well-defined and bounded.

[0202] Conventional techniques for enforcing quality of service preferences in an interconnected network can also be used to implement a fast path for the flow setup packet.

[0203] Figure 27 Embodiments of fast path processing that reduce the probability of packet retransmission are described. In block 2702, a first packet of a flow is transmitted on a lower priority channel. In block 2704, a second packet of the flow is transmitted on a higher priority channel. The second packet sets up the flow at the destination. Additional packets of the flow are transmitted on the lower priority channel (block 2706). The second packet that sets up the flow traverses the network with lower latency than conventional methods, thereby reducing the likelihood that a flow setup packet arrives after the first packet of the flow is evicted by the history filter, and also enabling single pieces to be evicted earlier, reducing the necessary history filter size in the receiver.

[0204] Protocol features disclosed herein can be implemented in computing devices that utilize one or more graphics processing units (GPUs) and / or general purpose data processors (e.g., 'central processing units or CPUs'). An exemplary architecture that can be configured to perform the techniques disclosed herein on such a device will now be described.

[0205] The following description can use certain acronyms and abbreviations, which are described below:

[0206] •“DPC” means“Data Processing Cluster”;

[0207] •“GPC” means“General Purpose Processing Cluster”;

[0208] •“I / O” means“Input / Output”;

[0209] •“L1 cache” means“Level 1 cache”;

[0210] •“L2 cache” means“Level 2 cache”;

[0211] •“LSU” means“Load / Store Unit”;

[0212] •“MMU” means“Memory Management Unit”;

[0213] •“MPC” means“M-Pipe Controller”;

[0214] •“PPU” means“Parallel Processing Unit”;

[0215] •“PROP” means“Pre-Raster Operations Unit”;

[0216] •“ROP” means“Raster Operations”;

[0217] •“SFU” means“Special Function Unit”;

[0218] •“SM” means“Streaming Multi-Processor”;

[0219] •“Viewport SCC” means“Viewport Scaling, Clipping and Culling”;

[0220] •“WDX” means“Work Distribution Crossbar”;

[0221] •“XBar” means“Crossbar”.

[0222] Parallel Processing Unit

[0223] Figure 28A parallel processing unit (PPU) 2820 according to one embodiment is illustrated. In one embodiment, the parallel processing unit 2820 is a multi-threaded processor implemented on one or more integrated circuit devices. The parallel processing unit 2820 is a latency-hiding architecture designed for parallel processing of many threads. A thread (i.e., an execution thread) is an instantiation of a set of instructions configured to be executed by the parallel processing unit 2820. In one embodiment, the parallel processing unit 2820 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device, such as a liquid crystal display (LCD) device. In other embodiments, the parallel processing unit 2820 may be used to perform general-purpose computing. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that this processor is illustrated for illustrative purposes only, and any processor may be used to supplement and / or replace this processor.

[0224] One or more parallel processing unit 2820 modules can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. The parallel processing unit 2820 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulation, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0225] like Figure 28 As shown, the parallel processing unit 2820 includes an I / O unit 2802, a front-end unit 2804, a scheduler unit 2808, a job distribution unit 2810, a hub 2806, a crossbar switch 2814, one or more general-purpose processing cluster 2900 modules, and one or more memory partitioning unit 3000 modules. The parallel processing unit 2820 can be interconnected to a host processor or other parallel processing unit 2820 modules via one or more high-speed NVLink 2816 interconnects. Embodiments of the protocols described herein can be implemented for communication between the various components of the parallel processing unit 2820 via NVLink 2816 and / or crossbar switch 2814.

[0226] The parallel processing unit(s) 2820 can be connected to the host processor(s) or other peripheral devices via an interconnect 2818. The parallel processing unit(s) 2820 can also be connected to a local memory comprising a number of memory devices 2812. In one embodiment, the local memory can include a number of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem in which multiple DRAM dies are stacked and accessed via a memory controller. The memory 2812 can include logic to configure the parallel processing unit(s) 2820 to perform aspects of the techniques disclosed herein.

[0227] The NVLink 2816 interconnect enables the system to scale with additional parallel processing unit(s) 2820 modules combined with one or more CPUs, supports cache coherency between the parallel processing unit(s) 2820 modules and CPUs, and host CPU offload. Data and / or commands can be sent by the NVLink 2816 to other units of the parallel processing unit(s) 2820 or from the parallel processing unit(s) 2820 by the hub 2806, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In conjunction with the various internal data buses, also shown are shared function units 2814, which are shared by the parallel processing unit(s) 2820 and the CPU(s). For example, shared function units 2814 can include copy engines, video encoders, video decoders, and / or other shared function units. Figure 32 The NVLink 2816 is described in further detail.

[0228] The I / O unit 2802 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) over the interconnect 2818. The I / O unit 2802 can communicate directly with the host processor via the interconnect 2818 or via one or more intermediate devices such as a memory bridge. In one embodiment, the I / O unit 2802 can communicate with one or more other processors, such as the parallel processing unit(s) 2820 module, via the interconnect 2818. In one embodiment, the I / O unit 2802 implements a Peripheral Component Interconnect Express (PCIe) interface for communications over a PCIe bus, and the interconnect 2818 is a PCIe bus. In alternative embodiments, the I / O unit 2802 can implement another type of known interface for communicating with external devices.

[0229] The I / O unit 2802 decodes packets received via the interconnect 2818. In one embodiment, the packets represent commands configured to cause the parallel processing unit 2820 to perform various operations. The I / O unit 2802 sends decoded commands, as specified by the commands, to various other units of the parallel processing unit 2820. For example, some commands can be sent to the front-end unit 2804. Other commands can be sent to the hub 2806 or to various other units such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, the I / O unit 2802 is configured to route communications between and among various logical units of the parallel processing unit 2820.

[0230] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload for processing by the parallel processing unit 2820. The workload can include a number of instructions and data to be processed by those instructions. The buffer is a region of memory that is accessible (e.g., read / write) by both the host processor and the parallel processing unit 2820. For example, the I / O unit 2802 can be configured to access the buffer in a system memory connected to the interconnect 2818 via memory requests transmitted over the interconnect 2818. In one embodiment, the host processor writes the command stream to the buffer and then sends a pointer to a start of the command stream to the parallel processing unit 2820. The front-end unit 2804 receives the pointer to the one or more command streams. The front-end unit 2804 manages the one or more streams, reading commands from the streams and forwarding the commands to various units of the parallel processing unit 2820.

[0231] The front-end unit 2804 is coupled to a scheduler unit 2808, which is configured to schedule various threads for execution via the general processing clusters 2900. The scheduler unit 2808 is configured to track state information related to various tasks managed by the scheduler unit 2808. The state can indicate which task is assigned to which general processing cluster 2900, whether the task is active or inactive, a priority associated with the task, etc. The scheduler unit 2808 manages execution of a plurality of tasks on the one or more general processing clusters 2900.

[0232] The scheduler unit 2808 is coupled to a work distribution unit 2810, which is configured to dispatch tasks for execution on the general processing cluster 2900 modules. The work distribution unit 2810 can track a number of scheduled tasks received from the scheduler unit 2808. In one embodiment, the work distribution unit 2810 manages a pending task pool and an active task pool for each general processing cluster 2900 module. The pending task pool can include a number of slots (e.g., 32 slots) that contain tasks that are assigned to be processed by a particular general processing cluster 2900. The active task pool can include a number of slots (e.g., 4 slots) for tasks that are actively being processed by a general processing cluster 2900 module. When a general processing cluster 2900 completes execution of a task, the task is evicted from the active task pool for the general processing cluster 2900, and one of the other tasks from the pending task pool is selected and dispatched for execution on the general processing cluster 2900. If the active task on a general processing cluster 2900 has idled, e.g., while waiting for a data dependency to be resolved, then the active task can be evicted from the general processing cluster 2900 and returned to the pending task pool, and another task in the pending task pool is selected and dispatched to the general processing cluster 2900 for execution.

[0233] The work distribution unit 2810 communicates with one or more general processing cluster 2900 modules via a crossbar 2814. The crossbar 2814 is an interconnect network that couples many of the units of the parallel processing unit 2820 to other units of the parallel processing unit 2820. For example, the crossbar 2814 can be configured to couple the work distribution unit 2810 to a particular general processing cluster 2900. Although not explicitly shown, one or more other units of the parallel processing unit 2820 can also be connected to the crossbar 2814 via the hub 2806.

[0234] Tasks are managed by the scheduler unit 2808 and dispatched to the general processing clusters 2900 by the work distribution unit 2810. The general processing clusters 2900 are configured to process tasks and generate results. The results can be consumed by other tasks within the general processing clusters 2900, routed to different general processing clusters 2900 via the crossbar 2814, or stored in the memory 2812. The results can be written to the memory 2812 via a memory partition unit 3000 module, which implements a memory interface for reading from and writing to the memory 2812. The results can be sent over the NVLink 2816 to another parallel processing unit 2820 or CPU. In one embodiment, the parallel processing unit 2820 includes a number U of memory partition units 3000 modules equal to the number of independent and distinct memory 2812 devices coupled to the parallel processing unit 2820. The memory partition unit 3000 modules will be described in more detail below in conjunction with FIG. 30.Figure 30 The memory partition unit 3000 is described in more detail.

[0235] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API), enabling one or more applications executing on the host processor to schedule operations for execution on the parallel processing unit 2820. In one embodiment, multiple computing applications are executed concurrently by the parallel processing unit 2820, and the parallel processing unit 2820 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. Applications can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the parallel processing unit 2820. The driver kernel outputs the tasks to one or more streams being processed by the parallel processing unit 2820. Each task may include one or more sets of associated threads, referred to herein as a warp. In one embodiment, a warp includes 32 associated threads that can execute in parallel. A cooperating thread can refer to multiple threads that include instructions for performing tasks and can exchange data via shared memory. Figure 31 A more detailed description of threads and cooperative threads.

[0236] Figure 29 An embodiment is shown. Figure 28 The general-purpose processing cluster 2900 consists of parallel processing units 2820. For example... Figure 29 As shown, each general-purpose processing cluster 2900 includes multiple hardware units for processing tasks. In one embodiment, each general-purpose processing cluster 2900 includes a pipeline manager 2902, a pre-raster operation unit 2904, a raster engine 2908, a job distribution crossbar switch 2914, a memory management unit 2916, and one or more data processing clusters 2906. It should be understood that... Figure 29 The general-purpose processing cluster 2900 can include alternatives Figure 29 Other hardware units of the unit shown or excluding Figure 29 Other hardware units besides the unit shown.

[0237] In one embodiment, the operations of general processing cluster 2900 are controlled by pipeline manager 2902. Pipeline manager 2902 manages configuration of one or more data processing cluster 2906 modules for processing tasks assigned to general processing cluster 2900. In one embodiment, pipeline manager 2902 can configure at least one of the one or more data processing cluster 2906 modules to implement at least a portion of a graphics rendering pipeline. For example, data processing cluster 2906 can be configured to execute vertex shader programs on a programmable streaming multi-processor 3100. Pipeline manager 2902 can also be configured to route packets received from work distribution unit 2810 to appropriate logical units within general processing cluster 2900. For example, some packets can be routed to fixed function hardware units in pre-raster operations unit 2904 and / or raster engine 2908, while other packets can be routed to data processing cluster 2906 modules for processing by geometry engine 2912 or streaming multi-processor 3100. In one embodiment, pipeline manager 2902 can configure at least one of the one or more data processing cluster 2906 modules to implement a neural network model and / or compute pipeline.

[0238] Pre-raster operations unit 2904 is configured to route data generated by raster engine 2908 and data processing cluster 2906 modules to a raster operations (ROP) unit for Figure 30 are described in greater detail. Pre-raster operations unit 2904 can also be configured to perform optimizations for color blending, organize pixel data, perform address translations, and the like.

[0239] Raster engine 2908 includes a number of fixed function hardware units that are configured to perform various raster operations. In one embodiment, raster engine 2908 includes a setup engine, a coarse raster engine, a cull engine, a clip engine, a fine raster engine, and a tile aggregation engine. The setup engine receives transformed vertices and generates a plane equation associated with a geometric primitive defined by the vertices. The plane equation is sent to the coarse raster engine to generate coverage information (e.g., an x, y coverage mask for a tile) for the geometric primitive. The output of the coarse raster engine is sent to the cull engine where fragments associated with primitives that fail a z-test are culled and sent to the clip engine where fragments that are outside a view frustum are clipped. Those fragments that survive clipping and culling can be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equation generated by the setup engine. The output of raster engine 2908 includes, for example, fragments to be processed by a fragment shader implemented within data processing cluster 2906.

[0240] Each data processing cluster 2906 included in the general processing cluster 2900 includes an M-pipe controller 2910, a primitive engine 2912, and one or more streaming multiprocessors 3100 modules. The M-pipe controller 2910 controls the operation of the data processing cluster 2906, routing packets received from the pipeline manager 2902 to the appropriate unit in the data processing cluster 2906. For example, a packet associated with a vertex can be routed to the primitive engine 2912, which is configured to fetch vertex attributes associated with the vertex from the memory 2812. Conversely, a packet associated with a shader program can be sent to the streaming multiprocessor 3100.

[0241] The streaming multiprocessor 3100 includes a programmable streaming processor configured to process tasks represented by a plurality of threads. Each streaming multiprocessor 3100 is multithreaded and configured to execute a plurality of threads (e.g., 32 threads) from a particular thread group simultaneously. In one embodiment, the streaming multiprocessor 3100 implements a single-instruction, multiple-data (SIMD) architecture wherein each thread in the thread group (e.g., a thread warp) is configured to process a different data set based on the same set of instructions. All threads in the thread group execute the same instructions. In another embodiment, the streaming multiprocessor 3100 implements a single-instruction, multiple-thread (SIMT) architecture wherein each thread in the thread group is configured to process a different data set based on the same set of instructions, but wherein individual threads in the thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each thread warp, enabling thread warps and serial execution within a thread warp to be concurrent when the threads within the thread warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, enabling equal concurrency among all threads within and between thread warps. When execution state is maintained for each individual thread, threads executing the same instructions can be converged and executed in parallel for maximum efficiency. The following discussion focuses on an embodiment in which the execution state is maintained for each individual thread. Figure 31 The streaming multiprocessor 3100 is described in more detail.

[0242] The memory management unit 2916 provides an interface between the general processing cluster 2900 and the memory partition unit 3000. The memory management unit 2916 can provide translations of virtual addresses into physical addresses, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit 2916 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses into physical addresses.

[0243] Figure 30 A memory partition unit 3000 of the parallel processing unit 2820 is shown according to one embodiment. As shown, the memory partition unit 3000 includes a memory management unit 2916, a memory partition scheduler 3004, and a memory 2812. The memory management unit 2916 provides an interface between the general processing cluster 2900 and the memory 2812. The memory management unit 2916 can provide translations of virtual addresses into physical addresses, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit 2916 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses into physical addresses. Figure 28 The memory partition unit 3000 of the parallel processing unit 2820 is shown according to one embodiment. As shown, the memory partition unit 3000 includes a memory management unit 2916, a memory partition scheduler 3004, and a memory 2812. The memory management unit 2916 provides an interface between the general processing cluster 2900 and the memory 2812. The memory management unit 2916 can provide translations of virtual addresses into physical addresses, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit 2916 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses into physical addresses.Figure 30 As shown, memory partition unit 3000 includes raster operations unit 3002, a level two cache 3004, and a memory interface 3006. Memory interface 3006 is coupled to memory 2812. Memory interface 3006 can implement a 32, 64, 128, 1024-bit data bus, etc. for high-speed data transfer. In one embodiment, parallel processing unit 2820 incorporates U memory interface 3006 modules, one for each pair of memory partition unit 3000 modules, with each pair of memory partition unit 3000 modules connected to a corresponding memory 2812 device. For example, parallel processing unit 2820 can be connected to up to Y memory 2812 devices, such as high bandwidth memory stacks or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent storage.

[0244] In one embodiment, memory interface 3006 implements an HBM2 memory interface and Y is equal to half of U. In one embodiment, the HBM2 memory stacks are located on the same physical package as parallel processing unit 2820, providing significant power and area savings compared to a conventional GDDR5 SDRAM system. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, with the HBM2 stack including two 128-bit channels per die for a total of 8 channels and a data bus width of 1024 bits.

[0245] In one embodiment, memory 2812 supports single error correction double error detection (SECDED) error correcting code (ECC) to protect data. For compute applications that are sensitive to data corruption, the ECC provides higher reliability of the data. In a large cluster computing environment, parallel processing unit 2820 modules process very large data sets and / or run applications for an extended period of time, reliability is especially important.

[0246] In one embodiment, parallel processing unit 2820 implements a multi-level memory hierarchy. In one embodiment, memory partition unit 3000 supports a unified memory to provide a single unified virtual address space for the CPU and parallel processing unit 2820 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of access to memory locations by the parallel processing unit 2820 is tracked, with memory pages that are frequently accessed being moved to faster storage, and less frequently accessed pages being moved to slower storage. In one embodiment, NVLink 2816 supports address translation services, which allow the parallel processing unit 2820 to directly access page tables stored in host memory by the CPU.

[0247] In one embodiment, the copy engine transfers data between the plurality of parallel processing units 2820 modules or between a parallel processing unit 2820 module and the CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 3000 can then service the page fault, map the address into a page table, after which the copy engine can perform the transfer. In a conventional system, memory is fixed (e.g., non-paged) for multiple copy engine operations between multiple processors, which significantly reduces the available memory. Due to the hardware page fault, an address can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0248] Data from memory 2812 or other system memory can be fetched by the memory partition unit 3000 and stored in a level two cache 3004, which is on-chip and shared between the various general processing cluster 2900 modules. As shown, each memory partition unit 3000 includes a portion of the level two cache 3004 that is associated with a corresponding memory device 2812. Lower level caches can then be implemented in multiple units within the general processing cluster 2900 modules. For example, each streaming multi-processor 3100 module can implement an LI cache. The LI cache is a private memory dedicated to a particular streaming multi-processor 3100. Data from the level two cache 3004 can be fetched and stored in each LI cache for processing in the functional units of the streaming multi-processor 3100 module. The level two cache 3004 is coupled to a memory interface 3006 and the crossbar 2814.

[0249] The raster operations unit 3002 performs graphics raster operations related to pixel colors such as color compression, pixel blending, etc. The raster operations unit 3002 also implements depth testing with the raster engine 2908, receiving a depth for a sample location associated with a pixel fragment from the culling engine of the raster engine 2908. The depth for the sample location associated with the fragment is tested against a corresponding depth in a depth buffer. If the fragment passes the depth test for the sample location, the raster operations unit 3002 updates the depth buffer and sends the results of the depth test to the raster engine 2908. It will be understood that the number of memory partition unit 3000 modules can be different from the number of general processing cluster 2900 modules, and thus each raster operations unit 3002 can be coupled to each general processing cluster 2900 module. The raster operations unit 3002 tracks the packets received from different general processing cluster 2900 modules and determines to which general processing cluster 2900 the results generated by the raster operations unit 3002 are routed through the crossbar 2814. Although in the embodiment shown in FIG. 29, the raster operations unit 3002 is shown as a separate unit from the general processing cluster 2900 modules, it will be understood that the raster operations unit 3002 can be implemented as part of the general processing cluster 2900 modules. Figure 30The raster operation unit 3002 is included within the memory partition unit 3000, but in other embodiments, the raster operation unit 3002 may be located outside the memory partition unit 3000. For example, the raster operation unit 3002 may reside in the general-purpose processing cluster 2900 or another unit.

[0250] Figure 31 An embodiment is shown. Figure 29 The streaming multiprocessor 3100. For example... Figure 31 As shown, the streaming multiprocessor 3100 includes an instruction cache 3102, one or more scheduler units 3104 modules (e.g., scheduler unit 2808), a register file 3108, one or more processing cores 3110 modules, one or more special function units 3112 modules, one or more load / store units 3114 modules, an interconnect network 3116, and a shared memory / L1 cache 3118. Embodiments of the protocols described herein can be implemented for communication between the various components of the streaming multiprocessor 3100 via the interconnect network 3116.

[0251] As described above, the work dispatch unit 2810 dispatches tasks to be executed on the general-purpose processing cluster 2900 module of the parallel processing unit 2820. Tasks are assigned to a specific data processing cluster 2906 within the general-purpose processing cluster 2900, and if the task is associated with a shader program, it can be assigned to the streaming multiprocessor 3100. The scheduler unit 2808 receives tasks from the work dispatch unit 2810 and manages the instruction scheduling of one or more thread blocks assigned to the streaming multiprocessor 3100. The scheduler unit 3104 schedules thread blocks to execute as thread bundles of parallel threads, wherein each thread block is assigned at least one thread bundle. In one embodiment, each thread bundle executes 32 threads. The scheduler unit 3104 can manage multiple different thread blocks, assign thread bundles to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (i.e., the core 3110 module, the special function unit 3112, and the load / store unit 3114) during each clock cycle.

[0252] A cooperation group is a programming model for organizing groups of communication threads that allows developers to express the granularity at which threads are communicating, enabling richer, more efficient parallel decomposition. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than a thread block and synchronize within the defined groups to achieve higher performance, design flexibility, and software reuse in the form of collective group-wide function interfaces.

[0253] Cooperation groups enable programmers to explicitly define thread groups at sub-block (e.g., as small as individual threads) and multi-block granularities and perform collective operations, such as synchronization across threads in a cooperation group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperation group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across a grid of thread blocks.

[0254] The dispatch 3106 units are configured to send instructions to one or more functional units within the scheduler unit 3104. In one embodiment, the scheduler unit 3104 includes two dispatch 3106 units that enable two different instructions from the same warp to be dispatched during each clock cycle. In alternative embodiments, each scheduler unit 3104 can include a single dispatch 3106 unit or additional dispatch 3106 units.

[0255] Each streaming multiprocessor 3100 includes a register file 3108 that provides a set of registers for the functional units of the streaming multiprocessor 3100. In one embodiment, the register file 3108 is partitioned among the various functional units such that each functional unit is allocated a dedicated portion of the register file 3108. In another embodiment, the register file 3108 is partitioned among different warps executed by the streaming multiprocessor 3100. The register file 3108 provides temporary storage for operands of the data

[0256] Each streaming multiprocessor 3100 includes L processing core 3110 modules. In one embodiment, the streaming multiprocessor 3100 includes a large number (e.g., 128, etc.) of distinct processing core 3110 modules. Each core 3110 can include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit that includes a floating-point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core 3110 module includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0257] The tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in the core 3110 module. Specifically, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on 4x4 matrices and performs matrix multiply and accumulate operations D = A'B + C, where A, B, C, and D are 4x4 matrices.

[0258] In one embodiment, the matrix multiply inputs A and B are 16-bit floating-point matrices, while the accumulate matrices C and D can be 16-bit floating-point or 32-bit floating-point matrices. The tensor cores operate on 16-bit floating-point input data and 32-bit floating-point accumulation. The 16-bit floating-point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating-point addition with other intermediate products of the 4x4x4 matrix multiplication. In practice, the tensor cores are used to perform much larger two-dimensional or higher-dimensional matrix operations built up from these smaller elements. APIs, such as the CUDA 9 C++ API, expose specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use the tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16x16 size matrices across all 32 threads of a warp.

[0259] Each streaming multiprocessor 3100 also includes M special-function units 3112 modules that perform special functions (e.g., attribute evaluations, inverse square root, etc.). In one embodiment, the special-function units 3112 modules can include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the special-function units 3112 modules can include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., 2D array of texture pixels) from memory 2812 and sample the texture map to produce sampled texture values for use in a shader program executed by the streaming multiprocessor 3100. In one embodiment, the texture map is stored in shared memory / L1 cache 3118. The texture unit implements texture operations, such as filtering operations using mip maps (i.e., texture maps of varying levels of detail). In one embodiment, each streaming multiprocessor 3100 includes two texture units.

[0260] Each streaming multiprocessor 3100 also includes N load / store units 3114 modules that implement load and store operations between shared memory / L1 cache 3118 and register file 3108. Each streaming multiprocessor 3100 includes an interconnect network 3116 that connects each functional unit to the register file 3108, as well as connects the load / store units 3114 to the register file 3108 and shared memory / L1 cache 3118. In one embodiment, the interconnect network 3116 is a crossbar that can be configured to connect any functional unit to any register in the register file 3108, as well as connect the load / store units 3114 modules to memory locations in the register file 3108 and shared memory / L1 cache 3118.

[0261] Shared memory / L1 cache 3118 is an on-chip memory array that allows data storage and communication between the streaming multiprocessors 3100 and the geometry engine 2912, as well as between threads in the streaming multiprocessors 3100. In one embodiment, shared memory / L1 cache 3118 includes 128 KB of storage capacity and is in the path from the streaming multiprocessors 3100 to the memory partition unit 3000. Shared memory / L1 cache 3118 can be used for cache reads and writes. One or more of shared memory / L1 cache 3118, level two cache 3004, and memory 2812 are backing stores.

[0262] Combining data cache and shared memory functionality into a single memory block provides optimal overall performance for both types of memory accesses. The capacity can be used by a program as a cache that does not use shared memory. For example, if the shared memory is configured to use half the capacity, then texture and load / store operations can use the remaining capacity. The integration within shared memory / L1 cache 3118 causes shared memory / L1 cache 3118 to function as a high-throughput pipe for streaming data and, at the same time, provide high-bandwidth and low-latency access to frequently-reused data.

[0263] When configured for general-purpose parallel computation, a simpler configuration can be used compared to graphics processing. Specifically, Figure 28 The illustrated fixed-function graphics processing units are bypassed, creating a simpler programming model. In the general-purpose parallel computation configuration, work distribution unit 2810 assigns and distributes thread blocks directly to data processing cluster 2906 modules. The threads in a block execute the same program, use the unique thread ID in the computation to ensure each thread generates a unique result, use streaming multiprocessor 3100 to execute the program and perform the computation, use shared memory / L1 cache 3118 to communicate between threads, and use load / store unit 3114 to read from and write to global memory through shared memory / L1 cache 3118 and memory partition unit 3000. When configured for general-purpose parallel computation, streaming multiprocessor 3100 can also write commands to scheduler unit 2808 that are usable to launch new work on data processing cluster 2906 modules.

[0264] Parallel processing unit 2820 can be included in a desktop computer, laptop computer, tablet computer, server computer, supercomputer, smart- phone (e.g., wireless, hand-held device), personal digital assistant (PDA), digital camera, vehicle, head-mounted display, hand-held electronic device, etc. In one embodiment, parallel processing unit 2820 is included on a single semiconductor die. In another embodiment, parallel processing unit 2820 is included in a system-on-a-chip (SoC) along with one or more other devices, such as additional parallel processing units 2820 modules, memory 2812, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.

[0265] In one embodiment, parallel processing unit 2820 can be included on a graphics card that includes one or more memory devices. The graphics card can be configured to interface with a PCIe slot on a motherboard of a desktop computer. In yet another embodiment, parallel processing unit 2820 can be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset of a motherboard.

[0266] Exemplary computing system

[0267] Systems with multiple GPUs and CPUs are being used across various industries as developers expose to and leverage greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of compute nodes are being deployed in data centers, research institutions, and supercomputers to tackle larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to be scaled to support this increased bandwidth.

[0268] Figure 32 This is based on the use of one embodiment. Figure 28 A conceptual diagram of a processing system 3200 implemented with parallel processing units 2820 is shown. The processing system 3200 includes a central processing unit 3206, a switch 3204, and each of multiple parallel processing unit 2820 modules, as well as corresponding memory 2812 modules. An NVLink 2816 provides a high-speed communication link between each parallel processing unit 2820 module. Although... Figure 32 A specific number of NVLink 2816 and interconnect 2818 connections are shown, but the number of connections to each parallel processing unit 2820 and central processing unit 3206 can vary. A switch 3204 interfaces between the interconnect 2818 and the central processing unit 3206. The parallel processing unit 2820, memory module 2812, and NVLink 2816 connections can reside on a single semiconductor platform to form the parallel processing module 1306. In one embodiment, the switch 3204 supports two or more protocols that interface between various different connections and / or links.

[0269] Implementations of the protocol described herein can be used (e.g., in NVLink 2816) to communicate between the various parallel processing units 2820 and / or between components of the parallel processing module 3202 and the central processing unit 3206 (e.g., in switch 3204).

[0270] In another embodiment (not illustrated), NVLinks 2816 provide one or more high-speed communication links between each parallel processing unit module (parallel processing unit 2820, parallel processing unit 2820, parallel processing unit 2820, and parallel processing unit 2820) and central processing unit 3206, and a switch 3204 interfaces between interconnect 2818 and each parallel processing unit module. The parallel processing unit modules, memory 2812 modules, and interconnect 2818 can be located on a single semiconductor platform to form a parallel processing module 3202. In yet another embodiment (not illustrated), interconnect 2818 provides one or more communication links between each parallel processing unit module and central processing unit 3206, and a switch 3204 interfaces between NVLinks 2816 to provide one or more high-speed communication links between each parallel processing unit module. In another embodiment (not illustrated), NVLinks 2816 provide one or more high-speed communication links between parallel processing unit modules and central processing unit 3206 through switch 3204. In yet another embodiment (not illustrated), interconnect 2818 provides one or more communication links between the various parallel processing unit modules directly. One or more NVLink 2816 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or on-die interconnects using the same protocol as NVLink 2816.

[0271] In the context of this specification, a single semiconductor platform can refer to a sole unitary integrated circuit that contains one or more processors and other resources. Alternatively, a single semiconductor platform can refer to a sole unitary package that contains one or more processors and other resources. Alternatively and more specifically, a single semiconductor platform can refer to a sole unitary chip or die that contains one or more processors and other resources. Of course, a single semiconductor platform can be formed from multiple, distinct chips or dies, which can be fabricated or manufactured separately, and which can be packaged individually or together in a single package. In this context, a chip or die can refer to a computing platform that can graphically represent a sole unitary integrated circuit, sole unitary package, or sole unitary chip or die.

[0272] In one embodiment, the signaling rate of each NVLink 2816 is 20 to 25 gigabits per second, and each parallel processing unit module includes six NVLink 2816 interfaces (as Figure 32As shown, each parallel processing unit module includes five NVLink 2816 interfaces. Each NVLink 2816 provides a 25 gigabit / second data transfer rate in each direction, with six lanes providing 300 gigabit / second. The NVLinks 2816 can be dedicated for CPU to GPU communication, as shown, or some combination of PPU to PPU and PPU to CPU. Figure 32

[0273] In one embodiment, the NVLink 2816 allows direct load / store / atomic operations from the central processing unit 3206 to the memory 2812 of each parallel processing unit module. In one embodiment, the NVLink 2816 supports coherency operations, allowing data read from the memory 2812 module to be stored in the cache hierarchy of the central processing unit 3206, reducing cache access latency of the central processing unit 3206. In one embodiment, the NVLink 2816 includes support for address translation services (ATS), allowing the parallel processing unit module to directly access page tables within the central processing unit 3206. One or more NVLinks 2816 can also be configured to operate in a low power mode.

[0274] Figure 33 An exemplary processing system 3300 is shown in which various architectures and / or functionality of various previous embodiments can be implemented. As shown, an exemplary processing system 3300 is provided that includes at least one central processing unit 3206 connected to a communication bus 3310. The communication bus 3310 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol(s). The exemplary processing system 3300 also includes a main memory 3302. Control logic (software) and data are stored in the main memory 3302, which can take the form of random access memory (RAM).

[0275] ​The example processing system 3300 also includes input devices 3308, parallel processing module 3202, and display devices 3306, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light-emitting diode), plasma display, or the like. User input can be received from input device 3308, e.g., keyboard, mouse, touchpad, microphone, or the like. Each of the foregoing modules and / or devices can even be located on a single semiconductor platform, e.g., an integrated circuit. Alternatively, various modules can be located on different semiconductor platforms, e.g., different chips or different portions of the same chip.

[0276] In addition, the example processing system 3300 can be coupled to a network (e.g., a telecommunications network, local area network (LAN), wireless network, wide area network (WAN) such as the Internet, peer-to-peer network, cable network, or the like) for communication purposes as desired by the user. The coupling can be a

[0277] The example processing system 3300 can also include secondary storage (not shown). The secondary storage includes, for example, a hard disk drive and / or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash drive, etc. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0278] Computer programs, or computer control logic algorithms, can be stored in the main memory 3302 and / or the secondary memory. Such computer programs, when executed, enable the example processing system 3300 to perform various functions. The primary memory 3302, the secondary memory, and / or any other storage are possible examples of computer-readable media.

[0279] The architectures and / or functions of the various preceding figures can be implemented in the context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, a special purpose system, and / or any other desired system. For example, the example processing system 3300 can take the form of a desktop computer, laptop computer, tablet computer, server computer, super computer, smart telephone (e.g., wireless, hand held device), personal digital assistant (PDA), digital camera, vehicle, head mounted display, hand held electronic device, mobile telephone device, television, workstation, game console, embedded system, and / or any other type of logic.

[0280] While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the preferred embodiments should not be limited by any of the above described exemplary embodiments, but should instead be defined in accordance with the following claims and their equivalents.

[0281] Graphics processing pipeline

[0282] Figure 34 According to one embodiment, by Figure 28 A conceptual diagram of a graphics processing pipeline 3400 implemented by a parallel processing unit 2820. In one embodiment, the parallel processing unit 2820 includes a graphics processing unit (GPU). The parallel processing unit 2820 is configured to receive commands specifying a shader program for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying multiple vertices of the primitive (e.g., in model space coordinates) and attributes associated with each vertex of the primitive. The parallel processing unit 2820 can be configured to process the graphics primitives to generate framebuffers (e.g., pixel data for each of the pixels in a display).

[0283] An application writes model data (e.g., a collection of vertices and attributes) for a scene into memory (such as system memory or memory 2812). The model data defines each of the objects that can be visible on a display. The application then makes an API call to the driver kernel, which requests the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands can reference different shader programs to be implemented on the streaming multiprocessor 3100 modules of the parallel processing unit 2820, including one or more of a vertex shader, a hull shader, a domain shader, a geometry shader, and a pixel shader. For example, one or more of the streaming multiprocessor 3100 modules can be configured to execute a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, different streaming multiprocessor 3100 modules can be configured to execute different shader programs concurrently. For example, a first subset of the streaming multiprocessor 3100 modules can be configured to execute a vertex shader program while a second subset of the streaming multiprocessor 3100 modules can be configured to execute a pixel shader program. The first subset of the streaming multiprocessor 3100 modules processes the vertex data to produce processed vertex data, which is written to the L2 cache 3004 and / or memory 2812. After the processed vertex data is rasterized (e.g., transformed from three-dimensional data into two-dimensional data in screen space) to produce fragment data, the second subset of the streaming multiprocessor 3100 modules executes the pixel shader to produce processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 2812. The vertex shader program and the pixel shader program can be executed concurrently, processing different data from the same scene in a pipelined fashion until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to a display controller to be displayed on a display device.

[0284] The graphics processing pipeline 3400 is an abstract flow diagram of processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is known, a pipeline architecture can more efficiently perform long latency operations by breaking the operation into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 3400 receives input data 3402 from a previous stage of the graphics processing pipeline 3400 to generate output data 3404 to a next stage of the graphics processing pipeline 3400. In one embodiment, the graphics processing pipeline 3400 can represent a graphics processing pipeline defined by an API. Alternatively, the graphics processing pipeline 3400 can be implemented in the context of the functionality and architecture of the previous figures and / or one or more of any subsequent figures. As such, the graphics processing pipeline 3400 can represent a graphics processing pipeline defined by an API. Alternatively, the graphics processing pipeline 3400 can be implemented in the context of the functionality and architecture of the previous figures and / or one or more of any subsequent figures.

[0285] As such, the graphics processing pipeline 3400 can represent a graphics processing pipeline defined by an API. Alternatively, the graphics processing pipeline 3400 can be implemented in the context of the functionality and architecture of the previous figures and / or one or more of any subsequent figures.Figure 34 As shown, graphics processing pipeline 3400 includes a pipeline architecture that includes a number of stages. These stages include, but are not limited to, a data assembly 3406 stage, a vertex shading 3408 stage, a primitive assembly 3410 stage, a geometry shading 3412 stage, a viewport SCC 3414 stage, a rasterization 3416 stage, a fragment shading 3418 stage, and a raster operations 3420 stage. In one embodiment, input data 3402 includes commands that configure the processing units to implement the stages of graphics processing pipeline 3400 as well as the geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by the stages. Output data 3404 can include pixel data (i.e., color data) that is copied to a frame buffer in memory or other type of surface data structure.

[0286] Data assembly 3406 stage receives input data 3402 that specifies vertex data for high-order surfaces, primitives, etc. Data assembly 3406 stage collects vertex data in temporary storage or a queue, such as by receiving a command from a host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to vertex shading 3408 stage for processing.

[0287] Vertex shading 3408 stage processes vertex data by performing a set of operations (e.g., a vertex shader or program) once for each of the vertices. A vertex can be specified, for example, as a 4-coordinate vector (e.g., <x, y, z, w>) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.). Vertex shading 3408 stage can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, vertex shading 3408 stage performs operations on vertex coordinates or other vertex attributes associated with a vertex. These operations typically include lighting operations (e.g., modifying a color attribute of a vertex) and transformation operations (e.g., modifying a coordinate space of a vertex). For example, a vertex can be specified using coordinates in an object coordinate space that are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to a world space or a normalized-device-coordinate (NDC) space. Vertex shading 3408 stage generates transformed vertex data that is passed to primitive assembly 3410 stage.

[0288] The primitive assembly 3410 stage collects vertices output by the vertex shading 3408 stage and groups the vertices into geometric primitives for processing by the geometry shading 3412 stage. For example, the primitive assembly 3410 stage can be configured to group every three consecutive vertices into a geometric primitive (e.g., a triangle) for passing to the geometry shading 3412 stage. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly 3410 stage passes the geometric primitives (e.g., the associated set of vertices) to the geometry shading 3412 stage.

[0289] The geometry shading 3412 stage processes the geometric primitives by performing a set of operations (e.g., a geometry shader or program) on the geometric primitives. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading 3412 stage can tessellate each geometric primitive into a finer grid of two or more geometric primitives for processing by the remainder of the graphics processing pipeline 3400. The geometry shading 3412 stage passes the geometric primitives to the viewport SCC 3414 stage.

[0290] In one embodiment, the graphics processing pipeline 3400 can sequentially perform processing operations within the stream multiprocessor and the vertex shading 3408 stage, the primitive assembly 3410 stage, the geometry shading 3412 stage, the fragment shading 3418 stage, and / or hardware / software associated therewith. Once the sequential processing operations are complete, in one embodiment, the viewport SCC 3414 stage can utilize the data. In one embodiment, primitive data processed by one or more of the stages in the graphics processing pipeline 3400 can be written into a cache (e.g., an LI cache, a vertex cache, etc.). In such a case, in one embodiment, the viewport SCC 3414 stage can access the data in the cache. In one embodiment, the viewport SCC 3414 stage and the rasterization 3416 stage are implemented as fixed function circuitry.

[0291] The viewport SCC 3414 stage performs viewport scaling, culling, and clipping of the geometry primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of a viewer looking at the scene and defines a view frustum that encloses the objects of the scene. The view frustum can include a viewing plane, a back plane, and four clipping planes. Any geometry primitive that is completely outside the view frustum can be culled (e.g., discarded) because it will not contribute to the final rendered scene. Any geometry primitive that is partially inside the view frustum and partially outside the view frustum can be clipped (e.g., transformed into a new geometry primitive that is enclosed within the view frustum). In addition, each geometry primitive can be scaled based on the depth of the view frustum. All potentially visible geometry primitives are then passed to the rasterization 3416 stage.

[0292] The rasterization 3416 stage converts the 3D geometry primitives into 2D fragments (e.g., that can be used for display, etc.). The rasterization 3416 stage can be configured to set up a set of plane equations using the vertices of the geometry primitive from which various attributes can be interpolated. The rasterization 3416 stage can also compute a coverage mask for a plurality of pixels that indicates whether one or more sample locations of the pixel intercept the geometry primitive. In one embodiment, a z-test can also be performed to determine whether the geometry primitive is occluded by other geometry primitives that have already been rasterized. The rasterization 3416 stage generates fragment data (e.g., interpolated vertex attributes associated with particular sample locations of each covered pixel) that is passed to the fragment shading 3418 stage.

[0293] The fragment shading 3418 stage processes the fragment data by performing a set of operations (e.g., a fragment shader or program) on each of the fragments. The fragment shading 3418 stage can generate pixel data (e.g., color values) for the fragment, such as by performing lighting operations or sampling texture maps using the interpolated texture coordinates of the fragment. The fragment shading 3418 stage generates pixel data that is sent to the raster operations 3420 stage.

[0294] The raster operations 3420 stage can perform various operations on the pixel data, such as performing an alpha test, a stencil test, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations 3420 stage has completed processing of the pixel data (e.g., output data 3404), the pixel data can be written to a render target, such as a frame buffer, a color buffer, etc.

[0295] It should be appreciated that one or more additional stages can be included in graphics processing pipeline 3400 in addition to or instead of one or more of the stages described above. Various implementations of an abstract graphics processing pipeline can implement different stages. Further, in some embodiments, one or more of the stages described above can be excluded from the graphics processing pipeline (such as the geometry shading 3412 stage). Other types of graphics processing pipelines are contemplated as being within the scope of the present disclosure. Further, any of the stages of graphics processing pipeline 3400 can be implemented by one or more specialized hardware units within a graphics processor (such as parallel processing unit 2820). Other stages of graphics processing pipeline 3400 can be implemented by programmable hardware units (such as streaming multiprocessors 3100 of parallel processing unit 2820).

[0296] Graphics processing pipeline 3400 can be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by an application program to generate graphics data for display. The device driver is a software program that includes a plurality of instructions that control the operation of parallel processing unit 2820. The API provides an abstraction for programmers that allows programmers to generate graphics data with specialized graphics hardware (such as parallel processing unit 2820) without requiring the programmer to utilize the specific instruction set of parallel processing unit 2820. The application program can include API calls that are routed to the device driver of parallel processing unit 2820. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least partially by initiating operations on parallel processing unit 2820 with an input / output interface between the CPU and parallel processing unit 2820. In one embodiment, the device driver is configured to utilize the hardware of parallel processing unit 2820 to implement graphics processing pipeline 3400.

[0297] Various programs can be executed within the parallel processing unit 2820 in order to implement various stages of the graphics processing pipeline 3400. For example, a device driver can initiate a kernel on the parallel processing unit 2820 to execute the vertex shading 3408 stage of the graphics processing pipeline 3400 on one streaming multi-processor 3100 (or multiple streaming multi-processor 3100 modules). The device driver (or initial kernel executed by the parallel processing unit 2820) can also initiate further kernels on the parallel processing unit 2820 to perform other stages of the graphics processing pipeline 3400, such as the geometry shading 3412 stage and the fragment shading 3418 stage. In addition, some of the stages of the graphics processing pipeline 3400 can be implemented on fixed function hardware, such as a rasterizer or a data assembler implemented within the parallel processing unit 2820. It will be appreciated that the results of a stage, prior to being processed by a subsequent stage, can be stored in memory such as memory 2824.

[0298] Figure 35 An exemplary data center 3500 is depicted in accordance with at least one embodiment. In at least one embodiment, data center 3500 includes, without limitation, a data center infrastructure layer 3502, a framework layer 3508, a software layer 3510, and an application layer 3520.

[0299] In at least one embodiment, as shown in Figure 35 the data center infrastructure layer 3502 can include a resource orchestrator 3504, grouped computing resources 3506, and node computing resources (“node C.R.s”) node C.R. 3526a, node C.R. 3526b, node C.R. 3526c,... node C.R. N), where “N” represents any integer, positive integer. In at least one embodiment, node C.R.s can include, without limitation, any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (“FPGAs”), graphics processors, etc.), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more of node C.R.s can be a server having one or more of above-described computing resources.

[0300] Embodiments of protocols described herein can be implemented for communication between various node computing resources, for example.

[0301] In at least one embodiment, the grouped computing resources 3506 may include individual groups of node CRs housed within one or more racks (not shown), or multiple racks housed in data centers (also not shown) located in different geographical locations. Individual groups of node CRs within the grouped computing resources 3506 may include grouped computing, networking, memory, or storage resources configurable or allocable to support one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches (in any combination).

[0302] In at least one embodiment, resource coordinator 3504 may be configured or otherwise control one or more node CRs and / or grouped computing resources 3506. In at least one embodiment, resource coordinator 3504 may include a software design infrastructure (“SDI”) management entity for data center 3500. In at least one embodiment, resource coordinator 3504 may include hardware, software, or some combination thereof.

[0303] In at least one embodiment, such as Figure 35 As shown, the framework layer 3508 includes, but is not limited to, a job scheduler 3512, a configuration manager 3514, a resource manager 3516, and a distributed file system 3518. In at least one embodiment, the framework layer 3508 may include a framework for software 3524 supporting the software layer 3510 and / or one or more applications 3522 supporting the application layer 220. In at least one embodiment, the software 3524 or application 3522 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 3508 may be, but is not limited to, a free and open-source software web application framework type that can utilize the distributed file system 3518 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(“Spark”). In at least one embodiment, the job scheduler 3512 can include a spare driver to facilitate scheduling of workloads supported by various tiers of the data center 3500. In at least one embodiment, a configuration manager 3514 can be capable of configuring different tiers, such as software tier 3510 and framework tier 3508, including Spark and a distributed file system 3518 for supporting large scale data processing. In at least one embodiment, a resource manager 3516 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 3518 and the distributed file system 3518. In at least one embodiment, clustered or grouped computing resources can include grouped computing resources 3506 at data center infrastructure layer 3502. In at least one embodiment, resource manager 3516 can coordinate with resource orchestrator 3504 to manage these mapped or allocated computing resources.

[0304] In at least one embodiment, software 3524 included in software tier 3510 can include software used by at least a portion of the node C.R.s of framework tier 3508, grouped computing resources 3506, and / or distributed file system 3518. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0305] In at least one embodiment, applications 3522 included in application tier 3520 can include one or more types of applications used by at least a portion of the node C.R.s of framework tier 3508, grouped computing resources 3506, and / or distributed file system 3518. At least one or more types of applications can include, but are not limited to, CUDA applications, 5G network applications, artificial intelligence applications, data center applications, and / or variations thereof.

[0306] In at least one embodiment, any of configuration manager 3514, resource manager 3516, and resource orchestrator 3504 can implement any number and type of self-modifying actions based on any number and type of data extracted in any technically feasible manner. In at least one embodiment, self-modifying actions can mitigate data center operators of data center 3500 making possible poor configuration decisions and possibly avoiding underutilization and / or poor performing portions of the data center.

[0307] List of Drawing Elements

[0308] 100 packet-based communication system

[0309] 102 source node

[0310] 104 retransmission buffer

[0311] 106 flow state

[0312] 108 same address filter

[0313] 110 shared memory

[0314] 112 shared memory

[0315] 114 history filter

[0316] 116 flow state

[0317] 118 replay buffer

[0318] 120 response reordering buffer

[0319] 122 request reordering buffer

[0320] 124 destination node

[0321] 126 source switch

[0322] 128 switch

[0323] 130 destination switch

[0324] 202 network

[0325] 2702 frame

[0326] 2704 frame

[0327] 2706 frame

[0328] 2802 I / O unit

[0329] 2804 front end unit

[0330] 2806 hub

[0331] 2808 scheduler unit

[0332] 2810 work distribution unit

[0333] 2812 memory

[0334] 2814 crossbar

[0335] 2816 NVLink

[0336] 2818 interconnect

[0337] 2820 parallel processing unit

[0338] 2900 general processing cluster

[0339] 2902 pipeline manager

[0340] 2904 Pre-raster operations unit

[0341] 2906 Data processing cluster

[0342] 2908 Raster engine

[0343] 2910 M-pipe controller

[0344] 2912 Primitives engine

[0345] 2914 Work distribution crossbar

[0346] 2916 Memory management unit

[0347] 3000 Memory partition unit

[0348] 3002 Raster operations unit

[0349] 3004 Level 2 cache

[0350] 3006 Memory interface

[0351] 3100 Streaming multiprocessor

[0352] 3102 Instruction cache

[0353] 3104 Scheduler unit

[0354] 3106 Dispatch

[0355] 3108 Register file

[0356] 3110 Core

[0357] 3112 Special function unit

[0358] 3114 Load / store unit

[0359] 3116 Interconnect network

[0360] 3118 Shared memory / L1 cache

[0361] 3200 Processing system

[0362] 3202 Parallel processing module

[0363] 3204 Switch

[0364] 3206 Central processing unit

[0365] 3300 Exemplary processing system

[0366] 3302 Main memory

[0367] 3304 network interface

[0368] 3306 display device

[0369] 3308 input device

[0370] 3310 communication bus

[0371] 3400 graphics processing pipeline

[0372] 3402 input data

[0373] 3404 output data

[0374] 3406 data assembly

[0375] 3408 vertex shading

[0376] 3410 primitive assembly

[0377] 3412 geometry shading

[0378] 3414 viewport SCC

[0379] 3416 rasterization

[0380] 3418 fragment shading

[0381] 3420 raster operations

[0382] 3500 data center

[0383] 3502 data center infrastructure layer

[0384] 3504 resource orchestrator

[0385] 3506 grouped computing resources

[0386] 3508 framework layer

[0387] 3510 software layer

[0388] 3512 job scheduler

[0389] 3514 configuration manager

[0390] 3516 resource manager

[0391] 3518 distributed file system

[0392] 3520 application layer

[0393] 3522 application program

[0394] 3524 software

[0395] 3526a Node C.R.

[0396] 3526b Node C.R.

[0397] 3526c Node C.R.

[0398] The various functional operations described herein can be implemented using logical name references or phrases that reflect the operations or functions. For example, an association operation can be performed by an “associator” or “correlator.” Likewise, switching can be by a “switcher,” selecting by a “selector,” and so on. “Logic” refers to machine memory circuitry and non-transitory machine-readable media including machine-executable instructions (software and firmware) and / or circuitry (hardware) that includes, by its material and / or material energy configuration, control and / or process signals, and / or settings and values (such as resistances, impedances, capacitances, inductances, current / voltage ratings, and so on) that can be applied to affect the operation of a device. Magnetic media, electronic circuitry, electrical and optical memory (both volatile and non-volatile), and firmware are examples of logic. Logic specifically excludes pure signals or software by itself (however, machine memory including software and thereby forming a material configuration is not excluded).

[0399] Within this disclosure, different entities (which can variously be referred to as “units,” “circuits,” other components, etc.) can be described or claimed as “configured” to perform one or more tasks or operations. Herein, use of the phrase “configured to” is used to generally refer to structures (i.e., physical things such as electronic circuits) that are arranged to perform one or more tasks during operation. More specifically, this phrase is used to indicate that a structure has been made, fixed, created, or formed to operate in a certain manner. A structure can be said to be “configured to” perform some task even if the structure is not currently being operated (e.g., a power supply is not connected to the structure). A “credit allocation circuit configured to allocate credits to a plurality of processor cores” is intended to cover, for example, an integrated circuit having circuitry that performs this function during operation, even if the integrated circuit in question is not currently being used (e.g., a power supply is not connected to the integrated circuit). Thus, an entity described as or recited as “configured to” perform some task refers to a physical thing, such as a device, circuit, memory storing program instructions executable to implement the task, etc. This phrase is not used herein to refer to intangibles.

[0400] The term “configured to” is not intended to mean “configurable to.” For example, an unprogrammed FPGA would not be considered to be “configured to” perform some specific function, even though it is “configurable to” perform that function after being programmed.

[0401] Structures described in the specification as “configured to” perform one or more tasks are expressly

[0402] As used herein, the term “based on” is used to describe one or more factors that affect a determination. This term is not exclusive, i.e., the determination can be based on additional factors not listed. For example, a determination can be based on one or more of the factors listed, and one or more other factors. In this context, the phrase “based on” is synonymous with the phrase “based at least in part on.” As used herein, the phrase “based on” is synonymous with the phrase “based at least in part on.”

[0403] As used herein, the phrase “in response to” describes one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors can influence or otherwise trigger the effect. That is, an effect can be in response to those factors alone, or in response to the specified factors and other, unspecified factors. Consider the phrase “perform A in response to B.” This phrase specifies B as a factor that triggers the performance of A. This phrase does not exclude the possibility that A can be performed in response to some other factor, e.g., C. This phrase is also intended to cover the embodiment where A is performed in response to B alone.

[0404] As used herein, the terms “first,” “second,” and the like, are used as labels for nouns that they precede, and do not imply any order or temporal sequence in connection with other objects or events described in the specification unless otherwise stated. For example, in a register file having eight registers, the terms “first register” and “second register” can be used to refer to any two of the eight registers, not just logical registers 0 and 1, for example.

[0405] The term “or” as used in a claim is used as an inclusive or and not as an exclusive or. For example, the phrase “at least one of x, y, or z” is interpreted as meaning x, y, or z individually, or any combination of x, y, and z.

[0406] As used herein, recitation of "and / or" with respect to two or more elements should be interpreted to mean only one element, or a combination of elements. For example, "element A, element B, and / or element C" can include just element A, just element B, just element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Also, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Also, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0407] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of the methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein commensurate with the purposes of the claim below.

[0408] Having described illustrative embodiments in detail, it will be apparent that modifications and variations are possible without departing from the scope of the inventive subject matter as defined in the claims. The scope of the inventive subject matter is not limited to the described embodiments but is instead defined by the following claims.

Claims

1. A transceiver for an unreliable and out-of-order switched packet network, the transceiver comprising: a transmitter; a receiver coupled to the transmitter through the switched packet network; logic to: determine, at the transmitter, a first condition that there is a dependency between an untransmitted packet and one or more already transmitted and not completed packets; tag, at the transmitter, the untransmitted packet with the same identifier and next sequence number as the one or more not completed packets on condition that the first condition is met; and determine, at the receiver, a second condition whether a second packet including the same identifier and different sequence number as a previously received packet is received; and establish, at the receiver, a packet stream with the transmitter on condition that the second condition is met.

2. The transceiver of claim 1, the receiver including logic to forward packets of the packet stream in an order defined by sequence numbers of the packets. the untransmitted packet has a same memory access address as a memory access address specified by the one or more already transmitted and not completed memory packets.

4. The transceiver of claim 1, further including logic to:

3. The transceiver of claim 1, wherein the dependency comprises: transmit, on condition that the first condition is met, the untransmitted packet to the receiver through a fast path of the switched packet network.

5. The transceiver of claim 1, the receiver further including logic to: utilize transaction ids of the previously received packet and the second packet as an identifier of the packet stream.

6. The transceiver of claim 1, further including logic to: perform a rollback operation to resynchronize the transmitter and the receiver in response to an error in the packet stream.

7. The transceiver of claim 1, further including logic to: establish the packet stream in response to receipt of the previously received packet and the second packet without an explicit handshake between the transmitter and the receiver to establish the packet stream.

8. The transceiver of claim 1, further including logic to: forward memory load command response packets in order to a processor's memory that is an original source of corresponding memory load command packets in response to a failed execution of the memory load command packets.

9. The transceiver of claim 1, further including logic to: detect, at the transmitter, multiple response packets to a same request packet sent to the receiver; and forward only a most recently received response packet of the multiple response packets to a processor's memory coupled to the transmitter.

10. A method of establishing a packet stream in an unreliable and out-of-order switched packet network, the method comprising: ​ ​ determining at a transmitter that an unsent memory access packet to be transmitted specifies access to a same memory address as specified by an outstanding memory access packet, and as a result marking the unsent memory access packet with a same transaction id and a later sequence number than the outstanding memory access packet; and detecting at a receiver that a logged packet and a second packet received after the logged packet have a same transaction id but different sequence numbers, and as a result establishing a same packet stream for the logged packet and the second packet.

11. The method of claim 10, wherein the further result of determining at the transmitter that the unsent memory access packet to be transmitted specifies access to a same memory address as specified by the outstanding memory access packet is to transmit the unsent memory access packet to the receiver with a higher priority than a priority at which the outstanding memory access packet is transmitted.

12. The method of claim 10, further comprising: establishing the packet stream based only on the logged packet and receipt of the second packet without performing an explicit handshake to establish the packet stream between the transmitter and the receiver prior to sending either packet.

13. The method of claim 10, further comprising: serializing in the packet stream transmission of memory load command packets and memory store command packets that specify the same memory address.

14. The method of claim 10, further comprising: re-performing a memory load operation for memory load command packets following a failed memory load command as a result of the failed memory load command packet being issued, by forwarding memory load command response packets in order.

15. A method of establishing a packet stream between a transmitter and a receiver in an unreliable and out-of-order switched packet network, the method comprising: transferring a first packet of the packet stream from the transmitter to the receiver at a first priority; transferring a second packet of the packet stream from the transmitter to the receiver at a second priority higher than the first priority; transferring additional packets of the packet stream from the transmitter to the receiver at the first priority after the second packet; and establishing at the receiver the existence of the packet stream as a result of receiving the second packet at the receiver such that the additional packets of the packet stream are recognized as belonging to the packet stream.

16. The method of claim 15, wherein the packet stream is established without a handshake procedure between the transmitter and the receiver.

17. The method of claim 15, further comprising: the packet stream including serialization of memory load command packets followed by memory store command packets in the packet stream.

18. The method of claim 15, further comprising: Tracking packets with a history filter in the receiver.

19. The method of claim 15, further comprising: Tracking packets with a same address filter in the transmitter.

20. The method of claim 15, further comprising: Performing a return mechanism in response to an error in the stream of packets.

Images