Devices that support both in-order and out-of-order transaction classes
Through the separate requester/responder architecture and virtual channel mechanism, the sorting of multiple transaction classes is supported, which solves the problem that transaction class sorting requirements in the prior art is difficult to meet, and improves system performance and reliability.
Patent Information
- Application Number
- CN202010376328.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-08
- Filing Date
- 2020-05-07
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-05-07
AI Technical Summary
Existing system interconnect communication protocols are difficult to effectively support the sorting requirements of multiple transaction classes, resulting in insufficient performance and reliability, especially in designs that eliminate the organizational sorting rules.
Adopting a separate requester/responder architecture, supporting unordered transaction classes and ordered transaction classes, independently selecting and transmitting packets through request queues and response queues, and using virtual channels to maintain the sorting requirements of committed transactions, allowing committed transaction requests and responses are tracked in flight status and are not transmitted until confirmation is completed.
Support for multiple transaction classes is realized, which reduces the amount of additional logic, allows public routing logic to support two types of transactions, improves the performance and reliability of the system, and meets the needs of different service rates and service quality.
Smart Images

Figure CN111917815B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to communications. Background Art
[0002] Some system interconnect communication protocols define various packet types that must be supported by components connected to the interconnect. For example, a protocol may define read requests, read responses, write requests, write confirmations, and various control packets. Conventional computer system interconnects such as PCIe partially provide correct behavior by specifying certain fabric ordering rules that are used to ensure that transactions are only visible in the expected order. To provide higher performance and reliability, some fabric interconnects such as Gen-Z are designed to eliminate these requirements. Summary of the invention
[0003] In one aspect, the present disclosure provides a system comprising: a requester, which is used to support an out-of-order transaction class and an in-order transaction class; and a responder, which is used to support an out-of-order transaction class and an in-order transaction class; wherein the requester is used to: select a submitted request to a destination from a request queue, wherein the submitted request is from the in-order transaction class and the request queue includes requests from the out-of-order transaction class; and transmit the submitted request to the destination; and the responder is used to: send the out-of-order transaction independently of the out-of-order transaction sent by the requester; select a first response to the destination from a response queue, wherein the first response is from the in-order transaction class and the response queue includes a second response from the out-of-order transaction class; receive a signal from the requester indicating that the submitted request is in-flight; and buffer the first response until a signal from the requester indicating that the submitted request is not in-flight is received.
[0004] In one aspect, the present disclosure provides an apparatus, comprising: a requester, which is used to support an out-of-order transaction class and an in-order transaction class, the requester comprising: a request queue; a request arbitrator, which is used to select a submitted transaction request to a destination from the request queue for transmission, wherein the submitted transaction request is of the in-order transaction class and the request sequence includes requests from the out-of-order transaction class; a request transceiver, which is used to transmit the submitted transaction request; and a tracker, which is used to track the end-to-end traversal of the submitted transaction request of the in-order transaction class; and a responder, the responder comprising: a response queue; a response arbitrator, which is used to select a first transaction response to the destination from the response queue, wherein the first transaction response is of the in-order transaction class and the response queue includes a second transaction response of the out-of-order transaction class; an interface to the requester, which is used to receive an indication of the end-to-end traversal of the submitted transaction request from the tracker; a response transceiver, which is used to buffer the first transaction response until the interface receives the indication, and when the indication is received, transmit the first transaction response.
[0005] In one aspect, the present disclosure provides a method for communication, comprising: a requester selects a submitted request to a destination from a request queue, the submitted request is from an ordered transaction class, and the request queue includes requests from an out-of-order transaction class; the requester transmits the submitted request to the destination; the requester tracks the in-flight status of the submitted request; a responder selects a first response to the destination from a response queue, the first response is from the ordered transaction class, and the response queue includes a second response from the out-of-order transaction class; the responder receives a signal from the requester indicating that the submitted request is in-flight; and the responder buffers the first response until the responder receives a signal from the requester indicating that the submitted request is not in-flight. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Certain examples are described in the following detailed description and with reference to the accompanying drawings, in which:
[0007] Figure 1 An example system is shown, the example system comprising a node located on a fabric including a requester and a responder;
[0008] Figure 2 An example device including a requester and a responder is shown;
[0009] Figure 3 An example device including a requester and a responder is shown;
[0010] Figure 4is a communication hop diagram illustrating an example series of communications between a first requester and responder at a first destination and a second requester and responder at a second destination; and
[0011] Figure 5 An example method of operation of a forked requester and responder is shown. DETAILED DESCRIPTION
[0012] Embodiments of the technology provide a system that supports multiple transaction classes by using a separate requester / responder architecture. A first transaction class may allow for loose ordering, where packets may pass over each other and arrive at a destination in an order different from the way they were sent from the source. A second transaction class has stricter ordering requirements, where certain types of packets cannot pass over other types of packets within a class. For example, the second transaction class may include a committed write request packet type and an uncommitted read response packet type, where an uncommitted read response cannot pass over a committed write request. For example, the first transaction class may be a set of core point-to-point Gen Z packet types, while the second transaction class may be a set of Gen Z PCIe compatible ordered packet types.
[0013] The node may have a split architecture where the requester and responder blocks inject packets onto the fabric independently of each other. The requester handles the injection of transaction requests such as read requests and write requests onto the fabric. The responder handles the injection of transaction responses such as read responses and write confirmations onto the fabric. In some embodiments, the requester and responder select packets from their respective queues independently of each other. The requester and responder inject packets of the first transaction class onto the fabric independently of each other.
[0014] For responses to the second transaction class that cannot bypass the submitted transaction request of the second class, the responder queries the requester to determine if there is a pending submitted transaction request bound for the same destination. If there is a pending submitted transaction, the responder buffers the selected response until the requester indicates that the pending submitted transaction is no longer pending. The architecture allows the responder to use the same queue, arbitration, and injection logic to handle transactions of both classes, thereby reducing the amount of additional logic required to support the two transaction classes. In addition, the architecture can allow transactions of both classes to be supported by common routing logic within the switch fabric that interconnects the requester and the responder.
[0015] Some embodiments support multiple virtual channels (VCs). A VC is a logical packet stream sent across a single physical link. For example, a component may utilize multiple VCs to enable packet authentication to provide different service rates, quality of service (QoS) levels, facilitate multipath and dynamic routing, prevent deadlock, or other features. Packets may be assigned to a VC by including a VC identifier as an information element associated with the packet.
[0016] In some embodiments, packets (request packets and response packets) of the second transaction class are transmitted on virtual channels separate from one or more virtual channels used to transmit packets of the first transaction class. In other embodiments, request packets of the second transaction class are transmitted on one or more virtual channels, and response packets of the second transaction class are transmitted on one or more different virtual channels. Both embodiments can use virtual channel information to maintain ordering requirements for submitted transaction requests of the second class. For example, in both embodiments, the fabric can apply the same route to all submitted request packets from source to destination.
[0017] In some embodiments, one or more virtual channels for packets of the second transaction class may be separate from one or more virtual channels for packets of the first transaction class. For example, this may allow packets of the second transaction class to have static routing through the fabric based on virtual channels without affecting the routing of packets of the first transaction class. In other embodiments, packets of both classes may use the same virtual channel. For example, read responses for two transaction classes may be sent on the same virtual channel, while write requests for one class may be sent on a different virtual channel than write requests for another class. As another example, write requests and read requests for both transaction classes may be sent on the same one or more virtual channels, and the fabric may use other information elements to route submitted transaction requests of the second transaction class in an orderly manner.
[0018] Figure 1 An example of multiple nodes connected via a packet switch fabric 108 is shown. Nodes 101, 118, 119, 120 can be a variety of different components. For example, a node may include a processor or system on chip (SOC) with an integrated memory or storage controller; a memory module such as a dynamic random access memory (DRAM), a non-volatile memory (NVM), or a storage class memory (SCM) module; a graphics card or other accelerator; a network controller; a switch; a digital signal processor; a custom application specific integrated circuit (ASIC) or field programmable gate array (FPGA) or other device that communicates using a memory semantic communication protocol. In the memory semantic protocol, the communication corresponds to a load / store processor architecture. Communication between components can be performed using read and write requests to specific addresses, such as at the byte address level. However, read and write requests are not necessarily limited to reading and writing data to memory or byte addressable storage. For example, in order to access an accelerator work queue or read from a completion queue, the accelerator can expose an address range that can be read and written.
[0019] In the example shown, the nodes are connected via a switch fabric 108. The switch fabric 108 may include multiple packet switches that connect components in a topology such as an all-to-all topology. In other examples, the nodes may be interconnected in a variety of different ways. For example, the nodes may be switch connected, directly connected, or daisy chained.
[0020] Each node component 101, 118, 119, 120 includes a component interface. Different components may have interface configurations. For example, nodes 101 and 120 have interfaces 105, 117 including requesters 102, 114 and responders 103, 115; node 118 has an interface 113 including a requester but no responder; and node 119 has an interface 116 including a responder but no requester. In this architecture, requesters 102, 114, 113 transmit requests to responders 103, 115, 116, and responders 103, 115, 116 transmit responses to received requests. For ease of explanation, each interface block (i.e., each requester 102, 114, 113 and each responder 103, 115, 116) is shown using a single link 106, 107, 109-112 to fabric 108. In some deployments, the single link may be logical rather than physical. For example, links 106 and 107 may be implemented by time multiplexing over a single duplex link, or another approach of providing dedicated logical links over a shared physical link may be employed.
[0021] In this example, the requester 102 and the responder 103 of the interface 105 support a first transaction class and a second transaction class.
[0022] The first transaction class includes a plurality of memory semantic packet types with a relaxed ordering model. For example, the first transaction class may include a read request, a read response, and a write request.
[0023] Read requests may be transmitted by requestors 102, 114, 113 to responders 103, 115, 116. These read requests are packets requesting data to be read out and returned from a memory address defined by the read request. Read responses are transmitted by responders 103, 115, 116 to requestors 102, 114, 113 to return the data requested by the corresponding read request. The first transaction class may include a single or multiple types of read requests. For example, the first transaction class may include different types of read requests to read N bytes of data starting from an indicated address, where N varies depending on the request type (e.g., a component may support a "Read 32" request to read 32 bytes of data starting from an address included in the request or a "Read 64" request to read 64 bytes of data starting from an address included in the request).
[0024] A write request is transmitted by a requestor and is a packet requesting that data be written to a specified address. A write request may include a starting memory address and a payload to be written to the memory at the starting memory address. In some embodiments, after executing the requested write, the write request may be confirmed by a confirmation packet transmitted by a responder. In other embodiments, the write request may not be confirmed by the receiving responder. In further embodiments, the architecture may support confirmed write requests and unconfirmed write requests within the first transaction class.
[0025] The second transaction class includes committed write requests, uncommitted requests, and responses.
[0026] A submitted request does not need to wait for a request completion response indicating the success or failure of the request. For example, when issuing a submitted write request, as soon as the requestor of the request sends the request on the fabric, the source of the request can consider the request completed without waiting for confirmation from the receiving responder. For example, once the requestor sends the submitted request, the component within the request source that causes the request to be transmitted (such as an application or operating system (OS) running on a node) can consider the submitted request completed.
[0027] Uncommitted requests require a completion response to indicate the success or failure of the request or to provide the requested data. For example, when the memory controller above issues an uncommitted read, the memory controller must wait for the requested data to be returned by the responder. As another example, the second transaction class may also include uncommitted write requests. The requestor that issues an uncommitted write request waits for write confirmation before acting in a manner that assumes the uncommitted write request is completed.
[0028] The second transaction class has an ordering model in which a submitted request can pass through unsubmitted requests and responses to the same destination and no request or response can pass through a submitted request to the same destination. For example, the ordering model can be a PCIe compatible ordering model.
[0029] In some embodiments, the first transaction class and the second transaction class may share a packet type or format. For example, a metadata tag, such as an information element in a write request packet header, may indicate whether the write request is a first transaction class or a second transaction class. Additionally, if the request is of the second class, the information element may indicate whether the request is a committed request or an uncommitted request.
[0030] return Figure 1, responder 103 sends transactions of the first class independently of transactions of the first class sent by requester 102. For example, responder 103 may perform queuing and priority arbitration decisions independently of queuing and priority arbitration performed by requester 102. For example, responder 103 may have a read response queue of pending read responses to send to requesters on other nodes 118-120. The responder may select a read response from the queue and transmit the read response over link 107 independently of the status of any requests that requester 102 may have in its own queue or that may be pending on fabric 108.
[0031] In some cases, the responders 103, 115, 116 are configured to send confirmations of the submitted transaction requests of the second transaction class to the requester. In some cases, these confirmations may be provided when the submitted transaction requests are received at the responders. In other cases, these confirmations may be provided after the requested transactions have been completed.
[0032] For example, in some embodiments, for a submitted write request, responder 115 sends an acknowledgement of receipt of the submitted write request regardless of whether the data in the request has been written to the responder's medium. This can be achieved when the written data is visible to subsequent reads before it is actually written to the responder's medium. This can be achieved, for example, if the responder can service subsequent reads from a write buffer while performing writes in parallel.
[0033] In other embodiments, responder 115 sends a receipt confirmation after the write request is completed. This may occur, for example, when the write is not visible to subsequent reads until the write is stored in the responder's medium. As another example, this may occur when the write needs to be durable and must be committed to a non-volatile data storage medium before the write can be considered durable. (In contrast, in examples where the write must be durable before subsequent reads but the responder includes a battery or capacitor to support a volatile cache for its non-volatile medium, a committed write confirmation may be sent before the responder commits the data to its non-volatile medium.)
[0034] The requester 102 uses the confirmation of the submitted transaction request to track the pendency of the submitted transaction request of the second transaction class on the fabric. In other words, the requester 102 indicates that the submitted transaction is pending from the time it is sent on the fabric 108 to the time the traversal confirmation is received from the destination. In some embodiments, the confirmation is not provided to the source of the request. In other words, since the requesting component can regard the submitted transaction request as completed when sending the submitted transaction request, it is not necessary to receive the confirmation of the submitted transaction request.
[0035] The system also includes a communication channel 104 between the responder 103 and the requester 102. The responder 103 uses the channel 104 to receive an indication from the requester 102 indicating whether the requester 102 has a pending previously committed transaction request of the second transaction class before sending the uncommitted transaction response of the second transaction class. For example, the channel 104 can be a bus or other communication link between the requester 102 and the responder 103, a port to a register set, table or other data structure maintained by the requester 102, or other channel.
[0036] In some embodiments, if there is a pending committed transaction of the second transaction class sent by the requester to the destination, the responder 104 does not send an uncommitted response to the destination. For example, when the responder 103 selects a transaction response from the second class whose destination is node X (i.e., one of nodes 118 or 120), the responder 103 queries the requester to determine whether there is a pending committed transaction request for node X. In other embodiments, if there is a pending committed transaction of the second transaction class sent by the requester to any destination, the responder 104 does not send an uncommitted response of the second transaction class to any destination.
[0037] In some cases, responder 103 delays sending uncommitted transaction responses until requester 102 does not have any pending previously committed transaction requests for the same destination as the uncommitted transaction. For example, responder 103 may buffer uncommitted transaction responses until receiving an indication that there are no previously committed transaction requests to the same destination. By waiting for these committed write requests to complete before issuing read responses, the technical solution eliminates the need for other parts of the fabric to manage the ordering of these flows. Since the committed request has already completed, the read response cannot pass the committed request.
[0038] Figure 2 An example device including a forked requester and a responder is shown. For example, device 201 may be such as Figure 1 Implementation of the apparatus of node 101 is described. The illustrated system components are shown as logic blocks that may be implemented in hardware as digital circuits, as firmware or software stored in a non-transitory medium such as flash memory or ROM, or a combination thereof.
[0039] Device 201 includes a system computer node having a requester 204 and a responder 205. For purposes of explanation, device 201 includes a CPU 202 including a memory controller 203 coupled to requester 204 and responder 205 and local memory 222. As described above, in other embodiments, device 201 may be a memory module such as a dynamic random access memory (DRAM), a non-volatile memory (NVM), or a storage class memory (SCM) module. In such embodiments, instead of CPU 202, the device may include a media controller having an interface to a memory or storage device and an interface to requester 204 and responder 205. In further embodiments, device 201 may be a graphics card or other accelerator, a network controller, a switch, a digital signal processor, a custom application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other device that communicates using a memory semantic communication protocol.
[0040] Device 201 includes a requester 204 that supports a first transaction class and a second transaction class. The requester includes a plurality of request transmission queues 206, 208, 210, 212. In this example, the requester supports four virtual channels (VCs) and maintains a separate transmission queue 206, 208, 210, 212 for each virtual channel. Although requester 204 is shown as having four VCs, embodiments may have a greater or lesser number of VCs, or may have only a single channel (which may be considered an implicit VC by it). In some embodiments, memory controller 203 or CPU 202 generates packets to be transmitted and places these packets directly into queues 206, 208, 210, 212, including assigning packets to their VCs. In other embodiments, CPU 202 transmits requests such as load or store requests to requester 204, responder 205, or an intermediary component (not shown) that then generates appropriate transaction packets to implement the CPU request.
[0041] Each queue 206, 208, 210, 212 is coupled to a queue arbiter 207, 209, 211, 213, respectively. The queue arbiters 207, 209, 211, 213 select packets from their respective queues 206, 208, 210, 212. For example, the queue arbiter 207 may apply one of various types of priority-based packet selection to select packets from the queues. Different arbiters 207, 209, 211, 213 may apply different priority techniques, or may apply the same technique. In some embodiments, some or all of the arbiters 207, 209, 211, 213 are absent, and the queues 206, 208, 210, 212 are first-in-first-out (FIFO) buffers.
[0042] VC arbiter 214 selects a packet to be transmitted from candidate packets from different VCs (i.e., from different queue arbiters 207, 209, 211, 213, or, if there is no queue arbitration, from the head of the FIFO buffer). VC arbiter 214 may apply a variety of different arbitration algorithms to select packets to be transmitted. For example, arbiter 214 may apply a round-robin technique, age-based priority, or a weighted version of one of these algorithms to select packets for transmission. The selected packet is sent to requester transceiver 215 for injection into fabric 224 on link 223 for transmission to the packet's destination.
[0043] Requester 204 also includes a plurality of receive queues 218, 219, 220, 221. Each receive queue 218, 219, 220, 221 corresponds to one of the VCs supported by requester 204. For example, receive queues 218, 219, 220, 221 may be FIFO buffers that store responses to requests previously sent by the requester.
[0044] Requester 204 also includes a tracker 216 that tracks the end-to-end traversal of the submitted transaction requests of the second transaction class. In this example, tracker 216 is coupled to transceiver 215 and inserted between VC arbiter 214 and transceiver 215. This allows tracker 216 to inspect packets before they are sent and track submitted transaction requests of the second transaction class. In other embodiments, tracker 216 can be located outside the main packet transmit and receive path. For example, tracker 216 can snoop the bus between VC arbiter 214 and transceiver 215 to determine when a submitted transaction request is transmitted.
[0045] Tracker 216 is further coupled to a data structure 217 maintained by it to track in-flight submitted transaction requests of the second transaction class. Tracker 216 can maintain data structures in various ways. For example, data structure 217 can be a table in which each entry corresponds to an in-flight submitted transaction request. For example, each entry can include a destination and ID of an in-flight submitted transaction. When transceiver 215 receives confirmation of a submitted transaction traversal, tracker 216 can clear the entry corresponding to the submitted transaction. As another example, each entry can include a destination ID and a counter that the tracker increments whenever a submitted transaction is sent to the corresponding ID and that the tracker decrements whenever a submitted transaction confirmation is received. In this example, when the counter reaches 0, the tracker clears the table entry.
[0046] As described above, a committed transaction request may be considered completed once it is transmitted. For example, an application running on CPU 202 may process a committed write request once it is injected onto fabric 224. Therefore, after tracker 216 uses the confirmation to track the traversal of the transaction request, the committed transaction traversal confirmation received by requestor 204 may be discarded. For example, if tracker 216 is in the path between receive queues 218, 219, 220, 221, tracker 216 may discard the committed transaction traversal confirmation without passing over these committed transaction traversal confirmations to receive queue management logic.
[0047] Device 201 also includes responder 205 to support the first transaction class and the second transaction class. Responder 205 includes multiple receive queues 238, 239, 240, 241. For example, receive queues 238, 239, 240, 241 can be FIFO buffers for storing received requests. In this example, the responder supports four virtual channels (VCs) and maintains separate receive queues 238, 239, 240, 241 for each VC. In some examples, some or all VCs supported by responder 205 are the same as those supported by requester 204. For example, VCs can be distinguished by ID numbers, and VCs supported by responders can have the same or overlapping ID number sets.
[0048] As described above, the responder receives requests from other requesters via fabric 224. These requests are queued in queues 238, 239, 240, 241. The request is then passed to the memory controller 203 which services the request, or passes the request to another component to be serviced. For example, when receiving a read request requesting data in local memory 222, the memory controller 203 retrieves the data from the memory and returns the data to the responder 205 for transmission to the request in a read response packet.
[0049] The responder 205 includes a plurality of response transmit queues 229, 231, 233, 235. In this example, the responder supports four virtual channels (VCs) and maintains a separate transmit queue 229, 231, 233, 235 for each virtual channel. In some implementations, the responder 205 may send a response packet of the second transaction class on a different VC than the requester 204 used to send the request packet of the second transaction class.
[0050] In some embodiments, the memory controller 203 or CPU 202 generates a response packet to be transmitted and places the response packet directly into a queue 229, 231, 233, 235, including assigning the packet to its VC. In another example, the memory controller 203 or other component returns data in raw or other format, and the responder 205 or intermediary component packages the data into an appropriate response packet and places the response into one of the queues 229, 231, 233, 235.
[0051] The responder 205 includes a pending transaction checker 225. The checker 225 includes a sideband interface 226 that enters the queues 229, 231, 233, 235. In some embodiments, the checker 225 monitors the response packets as they enter the queues 229, 231, 233, 235. In other embodiments, one or more of the VCs (and therefore, queues) are dedicated to responses of the second transaction class. In these embodiments, the checker 225 may only monitor the response packets placed in the dedicated queues.
[0052] In this example, if the monitored response packet is of the second transaction class, the checker 225 marks the packet as unavailable for transmission through the interface 242 and queries the requester 204 to receive an indication of an end-to-end traversal of the submitted transaction request. For example, the interface 242 can be coupled to the tracker 216 or the data structure 217. In one embodiment, the checker 225 can retrieve the response destination ID (i.e., the source of the request requiring the response) from the monitored packet and transmit the response destination ID to check whether there is a pending previously submitted transaction with the same destination node as the unsubmitted response. In another embodiment, the checker 225 checks whether there are any pending previously submitted transactions. Once the checker 225 receives a response indicating that there is no pending submitted transaction, the checker 225 clears the unavailable mark from the queued response packet.
[0053] In another example, a component generating a response packet for the second transaction class may create a packet with an unavailable flag. As another example, the unavailable flag is not used. Instead, the response packet for the second transaction class is marked as available only when the checker 225 determines from the requester 204 that a previously committed write transaction for the second transaction class has completed. In this example, the queue arbiter 230, 232, 234, 236 selects only from the second class of transactions with the available flag.
[0054] Each queue 229, 231, 233, 235 is coupled to a queue arbiter 230, 232, 234, 236, respectively. The queue arbiters 230, 232, 234, 236 select packets from their respective queues 229, 231, 233, 235. The queue arbiters 230, 232, 234, 236 may apply one of various types of priority-based packet selection to select packets from the queues. Different arbiters 230, 232, 234, 236 may apply different priority techniques, or may apply the same technique. In some embodiments, some or all of the arbiters 230, 232, 234, 236 do not exist, and the queues 230, 232, 234, 236 are first-in-first-out (FIFO) buffers.
[0055] The queue arbiter 230, 232, 234, 236 that manages the queue storing the responses of the second transaction class is configured not to select packets with an unavailable flag. Therefore, the applicable queue 230, 232, 234, 236 buffers the transaction response until the unavailable flag is cleared. As described above, after the checker receives an indication that a previously submitted transaction of the second transaction class (either any previously submitted transaction or a previously submitted transaction to the same destination, depending on the embodiment) has completed an end-to-end traversal, the unavailable flag of the response packet is cleared.
[0056] In an embodiment with a FIFO queue and no queue arbiter 229, 231, 233, 235, the checker 225 may remove packets from the queue or prevent them from being queued until receiving an indication that a previously submitted write request of the second class has completed an end-to-end traversal, rather than marking a response packet of the second transaction class.
[0057] VC arbiter 237 selects a packet for transmission from candidate packets from different VCs (i.e., from different queue arbiters 230, 232, 234, 236, or, if there is no queue arbitration, from the head of the FIFO buffer). VC arbiter 237 may apply a variety of different arbitration algorithms to select packets for transmission. For example, arbiter 237 may apply a round-robin technique, age-based priority, or a weighted version of one of these algorithms to select packets for transmission. The selected packet is sent to requester transceiver 227 for injection onto link 228 into fabric 224 for transmission to the packet's destination.
[0058] The responder 205 is also configured to transmit an end-to-end traversal confirmation to the external requester after receiving the submitted transaction request of the second transaction class from the external requester. In some embodiments, logic in the responder 205 (such as the checker 225) can generate and send the confirmation. For example, the checker 225 can be coupled to the receive queue and monitor the receive queue, and can generate the traversal confirmation. In this example, the checker 225 can then queue the traversal confirmation to one of the transmit queues 229, 231, 233, 235. In other embodiments, as described above with respect to Figure 1 As discussed, a traversal confirmation can be generated after completing the request.
[0059] Figure 3 An example device including a forked requester and responder is shown. For example, device 301 may be such as Figure 1 An embodiment of the device of Node 1 101 is described. The system components shown are shown as logic blocks that can be implemented in hardware as digital circuits, as firmware or software stored in a non-transitory medium such as flash memory or ROM, or a combination thereof.
[0060] Device 301 includes a system computer node having a requester 304 and a responder 305. For purposes of explanation, device 301 includes a CPU 302 including a memory controller 303 coupled to requester 304 and responder 305 and local memory 322. These elements may be as described with respect to CPU 202, memory controller 203, and local memory 222. As described above, in other embodiments, device 301 may be a memory module such as a dynamic random access memory (DRAM), non-volatile memory (NVM), or storage class memory (SCM) module, a graphics card or other accelerator, a network controller, a switch, a DSP, an ASIC, an FPGA, or other device that communicates by using a memory semantic protocol.
[0061] The device 301 also includes a requester 304 for supporting a first transaction class and a second transaction class. The requester includes a plurality of request transmission queues 306, 308, 310, 312. In this example, the requester supports four virtual channels (VCs) and maintains a separate transmission queue 306, 308, 310, 312 for each virtual channel. In this example, the queues 306, 308, 310, 312 are FIFO buffers. Each buffer is coupled to a VC arbiter 314, which selects a packet to be transmitted from the head of the buffer. As described with respect to the VC arbiter 214, the VC arbiter 314 can apply a variety of different arbitration algorithms. In other embodiments, the queues can be coupled to a queue arbiter that selects candidate packets from the queues 306, 308, 310, 312.
[0062] The requester 304 also includes a plurality of receive queues 318, 319, 320, 321. Each receive queue 318, 319, 320, 321 corresponds to one of the VCs supported by the requester 304. For example, the receive queues 318, 319, 320, 321 may be FIFO buffers storing responses to requests previously sent by the requester.
[0063] Requester 304 also includes a tracker 316 that tracks the end-to-end traversal of submitted transaction requests of the second transaction class. In this example, tracker 316 is coupled to transceiver 315 and is inserted between VC arbiter 314 and transceiver 315. This allows tracker 316 to inspect packets before they are transmitted and track submitted transaction requests of the second transaction class. In other embodiments, tracker 316 can be located outside the main packet transmission and reception path. For example, tracker 316 can monitor the bus between VC arbiter 314 and transceiver 315 to determine when submitted transaction requests are transmitted.
[0064] Tracker 316 is further coupled to a data structure 317 maintained by it to track in-flight submitted transaction requests of the second transaction class. Tracker 316 can maintain data structures in various ways. For example, data structure 317 can be a table in which each entry corresponds to an in-flight submitted transaction request. For example, each entry can include a destination and ID of an in-flight submitted transaction. When transceiver 315 receives confirmation of a submitted transaction traversal, tracker 316 can clear the entry corresponding to the submitted transaction. As another example, each entry can include a destination ID and a counter that the tracker increments whenever a submitted transaction is sent to the corresponding ID and that the tracker decrements whenever a submitted transaction confirmation is received. In this example, when the counter reaches 0, the tracker clears the table entry.
[0065] Device 301 also includes responder 305 to support the first transaction class and the second transaction class. Responder 305 includes multiple receive queues 338, 339, 340, 341. For example, receive queues 338, 339, 340, 341 can be FIFO buffers for storing received requests. In this example, the responder supports four virtual channels (VCs) and maintains separate receive queues 338, 339, 340, 341 for each VC. In some examples, some or all VCs supported by responder 305 are the same VCs as those supported by requester 304. For example, VCs can be distinguished by ID numbers, and VCs supported by responders can have the same or overlapping ID number sets.
[0066] As described above, the responder receives requests from other requesters via fabric 324. These requests are queued in queues 338, 339, 340, 341. The request is then passed to the memory controller 303 which services the request, or passes the request to another component to be serviced. For example, upon receiving a read request requesting data in local memory 322, the memory controller 303 retrieves the data from the memory and returns the data to the responder 305 for transmission to the request in a read response packet.
[0067] The responder 305 includes a plurality of response transmission queues 329, 331, 333, 335. In this example, the responder supports four virtual channels (VCs) and maintains a separate transmission queue 329, 331, 333, 335 for each virtual channel. In some embodiments, the responder 305 may send a response packet of the second transaction class on a different VC than the requester 304 used to send the request packet of the second transaction class.
[0068] In some embodiments, the memory controller 303 or CPU 302 generates response packets to be transmitted and places these response packets directly into the queues 329, 331, 333, 335, including assigning the packets to their VCs. In another example, the memory controller 303 or other component returns data in raw or other formats, and the responder 305 or intermediary component packages the data into an appropriate response packet and places the response into one of the queues 329, 331, 333, 335.
[0069] In this example, each transmit queue 329, 331, 333, 335 is a FIFO buffer coupled to a VC arbiter 337. In other examples, such as regarding Figure 2 327. As described above, the queues may be coupled to a queue arbiter. VC arbiter 337 selects a packet to be transmitted from the candidate packets at the head of FIFO buffers 329, 331, 333, 335. VC arbiter 337 may apply the various arbitration techniques described with respect to VC arbiter 227. The selected packet is sent to responder transceiver 327 for injection onto link 328 into ingress fabric 324 for transmission to the packet's destination.
[0070] The responder 305 also includes a queuer 325. The queuer 325 is coupled to an interface 342 to the requester 304 to receive an indication of an end-to-end traversal of a committed transaction request from a tracker. For example, the interface 342 may be coupled to the tracker 316 or the data structure 317. When the arbiter 337 selects a response of a second transaction class (such as an uncommitted read response) for transmission on the link 328, the queuer 325 checks whether there is a pending committed transaction sent by the requester 304. For example, the queuer 325 may check whether there is a pending committed transaction with the same destination as the uncommitted response. Alternatively, the queuer 325 may check whether there is any pending committed transaction to any destination.
[0071] Requester 305 also includes a transceiver 327 that transmits the selected transaction response based on the indication. For example, responder 305 can buffer the selected transaction response in buffer 326 while there are pending committed transactions that prevent transmission of the selected response. Buffer 326 can be sized to buffer multiple pending responses. For example, buffer 326 can be sized based on a typical round trip delay of committed transactions on the fabric and a packet transmission rate of the responder.
[0072] The responder 305 is also configured to transmit an end-to-end traversal confirmation to the external requester after receiving the submitted transaction request of the second transaction class from the external requester. In some embodiments, logic in the responder 305 (such as the transceiver 327 or the queuer 225) can generate and send the response. In other embodiments, other system components can generate a traversal response after the corresponding request is completed.
[0073] Figure 4 An example network hop diagram showing a series of example communications between a requester 402 and responder 401 at a first destination device A and a second requester 405 and responder 406 at a second destination device B. The communicating devices are interconnected by a switch network represented by switch 1 403 and switch 2 404 .
[0074] In this example, requester A 402 transmits a first type of write request (WR 1 407) and the second type of submitted write request (PWR 2 408). Responder B 406 performs the requested write and sends a 2 408 Send confirmation PWR-ACK 2 411. In various embodiments, responder B 406 may also send a WR 1 However, for the sake of clarity, Figure 3The confirmation is omitted in the WR. The transaction request of the first type is not required to maintain the ordering relationship with the transaction of the second type. In this example, switch 1 403 is in WR 1 Forward PWR before 407 2 408, thereby reversing the order of the transactions compared to their original transmission order.
[0075] Here, requester B 405 sends a second type of read request (RD 2 410) and the first type of read request (RD 1 409). Responder A 401 receives the read request and prepares to send a read response RD-RSP containing the requested data. 1 412 and RD-RSP 2 413.
[0076] When sending RD-RSP 2 Prior to 413, responder A 401 queries 414 requester A 402 to determine if there is a pending committed transaction to destination B. When requester A 402 receives query 414, it has not yet received a PWR-ACK. 2 411. Thus, in response to query 414, requester A 402 replies 415 that yes, there is a pending committed transaction to destination B. In response, responder A 401 buffers the RD-RSP 2 413 and does not send it. 1 In the case of 409, the transaction is of the first type, so responder A 401 will RD-RSP 1 412 is sent to requestor B 405 without checking whether there is a pending write request to target B.
[0077] In this embodiment, responder A 401 repeats query 416 to requester A 402 to determine if there is a pending committed transaction to destination B. Requester A 402 has received a PWR-ACK prior to query 416. 2 411. Therefore, requester A 402 responds 417 that there is no pending committed transaction to destination B. Responder A 401 then sends RD-RSP 2 is sent to Requester B 405. In an alternative embodiment, Requester A 402 delays sending a response to the query 414 until the write acknowledgement PWR-ACK has been submitted 2 In this embodiment, the lack of a response indicates that there is a pending committed write request of the second transaction class, and responder A 401 buffers the RD-RSP 2 , until after receiving a response.
[0078] Figure 5 The operation method of the device including the requester and the responder is shown. For example, Figure 1 Node 101, Figure 2 Equipment 201, Figure 3 Device 301 or Figure 3 The destination A can be as follows Figure 5 Operate as described.
[0079] The method includes block 501. Block 501 includes a requester selecting a submitted request to a destination from a request queue. As described above, a requester can request a request from a request queue independently of queuing and arbitration performed by a responder. The request queue can also include unsubmitted requests. Submitted requests are from an ordered transaction class, while unsubmitted requests may be from an unordered transaction class or an ordered transaction class. For example, as described above, a requester can support multiple transaction classes, including an ordered transaction class and an unordered transaction class. In an ordered transaction class, a submitted transaction request (e.g., a submitted write request) can pass over unsubmitted requests and unsubmitted responses (including completion or confirmation packets) to the same destination, and no request or response can pass over submitted requests to the same destination. For example, the sorting model can be a PCIe compatible sorting model. In an unordered transaction class, all requests and responses from one destination to another destination can pass over each other on the fabric.
[0080] The method also includes block 502. Block 502 includes the requester sending the submitted request to the destination. For example, block 502 may include the requester transmitting the submitted request on the first virtual channel. As described above, block 502 may also include transmitting the request independently of the response transmitted by the responder.
[0081] The method also includes block 503. Block 503 includes the requester tracking the in-flight status of the submitted request. For example, block 503 may also include the requester receiving an acknowledgment from the destination. Here, block 503 may include tracking the submitted request in-flight as the submitted request is sent to the destination, and in response to the acknowledgment, updating the in-flight status of the submitted request. For example, the requester may update the in-flight status of the submitted request by marking the request as completed in the data structure or removing the tracking data for the request from the data structure.
[0082] The method also includes block 504. Block 504 includes the responder selecting a first response to the destination from the response queue. As described above, the responder can select the first response from the response queue independently of the requester selecting a submitted request from the request queue. In some embodiments, the response queue can include a mix of responses from an out-of-order transaction class and an ordered transaction class.
[0083] The method also includes block 505. Block 505 includes the responder receiving a signal from the requester indicating that a submitted request is in flight. For example, block 505 may include the responder querying the requester to determine whether there is a submitted request in flight to the destination in response to selecting the first response. The received signal may be in response to the responder querying the requester.
[0084] The method also includes block 506. Block 506 includes the responder buffering the first response until the responder receives a signal from the requester indicating that the submitted request is not in flight. For example, the responder can repeat blocks 505 and 506 to repeatedly query the requester until the responder receives a signal that the submitted request is not in flight. As another example, the requester can delay sending a signal in response to the query until the submitted request is not in flight.
[0085] The method also includes block 507. Block 507 includes the responder transmitting the first response to the requester at the destination. For example, block 507 may include transmitting the response on a virtual channel different from the virtual channel used by the requester to transmit the submitted request. As another example, block 507 may include transmitting the response on the same virtual channel as the submitted request.
[0086] In the above description, many details are set forth to provide an understanding of the subject matter disclosed herein. However, embodiments herein may be implemented without some or all of these details. Other embodiments may include modifications and variations to the above details. The appended claims are intended to cover these modifications and variations.
Claims
1. A system comprising: Requester, which is used to support unordered transaction classes and ordered transaction classes; as well as Responders, which are used to support both unordered and ordered transactions; in The requester is used to: selecting a submitted request to a destination from a request queue, wherein the submitted request is from the in-order transaction class and the request queue includes requests from the out-of-order transaction class; as well as transmitting the submitted request to the destination; and The responder is used to: sending the out-of-order class transaction independently of an out-of-order class transaction sent by the requestor; selecting a first response to the destination from a response queue, wherein the first response is from the in-order transaction class and the response queue includes a second response from the out-of-order transaction class; receiving a signal from the requester indicating that the submitted request is in-flight; as well as The first response is buffered until a signal is received from the requestor indicating that the submitted request is not in flight.
2. The system of claim 1, wherein: The responder is further configured to receive, prior to sending an uncommitted transaction response for the ordered class, an indication from the requester indicating whether the requester has a pending previously committed transaction request for the ordered class; and The requester is further configured to provide the indication to the responder after receiving confirmation that the pending previously committed transaction request arrived at the first destination.
3. The system of claim 1, wherein: The responder is used to query the requester before sending an uncommitted transaction response of the ordered transaction class, and The indication is responsive to the inquiry.
4. The system of claim 2, wherein: The responder is configured to delay sending the uncommitted transaction response until after the requester does not have any pending previously committed transaction requests of the ordered transaction class for the same destination as the uncommitted transaction response.
5. The system of claim 2, wherein: The responder is configured to delay sending the uncommitted transaction response until after the responder does not have any pending previously committed transaction requests of the ordered class.
6. The system of claim 2, wherein: The responder is used to send a confirmation for the received submitted transaction request of the ordered class, and the confirmation is sent before the requested submitted transaction is completed.
7. The system of claim 2, wherein: The responder is configured to send the uncommitted transaction response on a separate virtual channel from the committed transaction request.
8. The system of claim 1, wherein: The responder is operable to select a response to send independently of a request selected to send by the requester, and to buffer selected uncommitted responses for the ordered transaction class until the indication is received.
9. An apparatus comprising: A requester, which is used to support an unordered transaction class and an ordered transaction class, and the requester includes: Request queue; a request arbiter for selecting a submitted transaction request to a destination from the request queue for transmission, wherein the submitted transaction request is of the in-order transaction class and the request queue includes requests from the out-of-order transaction class; a request transceiver for transmitting the submitted transaction request; and a tracker for tracking an end-to-end traversal of the submitted transaction requests of the ordered transaction class; and A responder, the responder comprising: Response queue; a response arbiter for selecting a first transaction response to the destination from the response queue, wherein the first transaction response is of the in-order transaction class and the response queue includes a second transaction response of the out-of-order transaction class; an interface to the requester for receiving an indication of an end-to-end traversal of the submitted transaction request from the tracker; A response transceiver is configured to buffer the first transaction response until the interface receives the indication, and transmit the first transaction response when the indication is received.
10. The device according to claim 9, wherein: The responder also includes: A buffer is provided for storing the transaction response when the transaction response is of the ordered transaction class and the indication indicates that a previously submitted transaction request of the ordered transaction class has not completed an end-to-end traversal.
11. The device according to claim 9, wherein: The responder also includes: A buffer is provided for storing the transaction response when the transaction response is of the ordered transaction class and the indication indicates that a previously submitted transaction request of the ordered transaction class addressed to the same destination node as the selected transaction response has not completed an end-to-end traversal.
12. The device according to claim 9, wherein: The transceiver is used to receive confirmation of an end-to-end traversal of submitted transaction requests of the ordered transaction class.
13. The device according to claim 9, wherein: The responder is used to transmit an end-to-end traversal confirmation to the external requester after receiving the submitted transaction request of the ordered transaction class from the external requester.
14. The device according to claim 9, wherein: The responder is configured to transmit the transaction response on a virtual channel different from the virtual channel used by the request transmitter to transmit the transaction request.
15. The device according to claim 9, wherein: The interface is for interrogating the tracker to receive the indication before the response arbitrator selects a transaction response of the ordered transaction class.
16. The device according to claim 9, wherein: The interface is for querying the tracker to receive the indication after the response arbitrator selects a transaction response of the ordered transaction class.
17. A method for communication, comprising: The requester selects a submitted request to a destination from a request queue, the submitted request being from an in-order transaction class, and the request queue including requests from an out-of-order transaction class; The requester transmits the submitted request to the destination; The requester tracks the in-flight status of the submitted request; a responder selecting a first response to the destination from a response queue, the first response being from the in-order transaction class, and the response queue including a second response from the out-of-order transaction class; The responder receives a signal from the requester indicating that the submitted request is in-flight; as well as The responder buffers the first response until the responder receives a signal from the requester indicating that the submitted request is not in flight.
18. The method according to claim 17, further comprising: The responder selects the first response from the response queue independently of the requester selecting the submitted request from the request queue.
19. The method according to claim 17, further comprising: The requester receives an acknowledgment from the destination and in response updates the in-flight status of the submitted request.
20. The method of claim 17, further comprising: In response to selecting the first response, the responder queries the requester to determine whether there is an in-flight submitted request of the ordered transaction class to the destination; wherein: The responder receives the signal in response to the responder interrogating the requester.
