Scheduling using decoupled grant and accept stages

Abstract

Techniques and apparatuses are described for scheduling using decoupled grant and accept stages. In example aspects, an arbiter (202) of a crossbar scheduler uses buffering to decouple the grant and accept stages (314 and 318) for scheduling. The buffering divides the scheduling operation into input scheduling (310) and output scheduling (312), which can be performed in parallel. With a decoupling circuit of the arbiter (202) providing the buffering, data flows in a pipeline fashion from an initiator (302) to a target (304) across at least two cycles. The decoupling of the grant and accept stages (314 and 318) provides additional flexibility in performing the input scheduling (310) using an input-first mode (322) or an output-first mode (324). By decoupling the grant and accept stages (314 and 318), the arbiter (202) can have a scalable architecture that supports high-speed and high-throughput scheduling.

Description

SCHEDULING USING DECOUPLED GRANT AND ACCEPT STAGESBACKGROUND

[0001] An electronic device can be implemented with a system-on-chip (SoC), which can provide many features of the electronic device. An example system-on-chip can include multiple subsystems, such as a central processing unit (CPU), a graphics processing unit (GPU), and / or an image processing unit (IPU). Technological advancements enable a system-on-chip to be designed with a larger quantity of subsystems to further expand feature capabilities of the electronic device. The larger quantity of subsystems, however, can increase a complexity for establishing communications between these subsystems.SUMMARY

[0002] Techniques and apparatuses are described for scheduling using decoupled grant and accept stages. In example aspects, an arbiter of a crossbar scheduler uses buffering to decouple the grant and accept stages for scheduling. The buffering divides the scheduling operation into two operations, which can be performed in parallel. A first operation involves input scheduling, which transfers data from an initiator to a decoupling circuit of the arbiter as part of the grant stage. A second operation involves output scheduling, which transfers data from the decoupling circuit to a target as part of the accept stage. With the decoupling circuit, data flows in a pipeline fashion from the initiator to the target across at least two cycles. The decoupling of the grant and accept stages provides additional flexibility in performing the input scheduling using an input-first mode or an output-first mode. By decoupling the grant and accept stages, the arbiter can have a scalable architecture that supports high-speed and high-throughput scheduling.

[0003] Aspects described below include a first method performed by an arbiter. The method includes performing, during a first cycle, input scheduling to transfer data from at least one initiator of multiple initiators to a decoupling circuit of the arbiter. The multiple initiators are coupled to multiple input ports of the arbiter. The method also includes performing, during a second cycle, output scheduling to transfer the data from the decoupling circuit to at least one target of multiple targets. The multiple targets are coupled to multiple output ports of the arbiter.

[0004] Aspects described below include a second method performed by an arbiter. The method includes performing, during a first cycle, output scheduling to transfer a first beat from a decoupling circuit of the arbiter to a first target. The method also includes performing, during the first cycle, input scheduling to transfer a second beat from a first initiator to a decoupling circuit of the arbiter.

[0005] Aspects described below also include an apparatus including a crossbar scheduler. The crossbar scheduler includes an arbiter configured to perform any of the described methods.

[0006] Aspects described below7include a computer-readable storage medium comprising computer-executable instructions that, responsive to execution by an arbiter, cause the arbiter to perform any one of the described methods.

[0007] Aspects described below also include a system with means for scheduling using decoupled grant and accept stages.BRIEF DESCRIPTION OF DRAWINGS

[0008] Apparatuses for and techniques for scheduling using decoupled grant and accept stages are described with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:FIG. 1 illustrates an example environment in which scheduling using decoupled grant and accept stages can be implemented;FIG. 2 illustrates an example implementation of a computing device that can implement aspects of scheduling using decoupled grant and accept stages;FIG. 3 illustrates an example operation of an arbiter capable of scheduling using decoupled grant and accept stages;FIG. 4 illustrates an example relationship between multiple initiators, multiple targets, and an arbiter capable of scheduling using decoupled grant and accept stages;FIG. 5 illustrates example internal and external communications of an arbiter for performing scheduling using decouple grant and accept stages;FIG. 6 illustrates example transactions between an initiator, an input scheduler, a skid buffer, an output scheduler, and a target for scheduling using decoupled grant and accept stages;FIG. 7. illustrates a flow of data from an initiator to a target based on scheduling using decoupled grant and accept stages:FIG. 8 illustrates a first example implementation of a decoupling circuit for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 9 illustrates a second example implementation of a decoupling circuit for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 10 illustrates a first example implementation of an arbiter for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 11-1 illustrates a second example implementation of an arbiter for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 11-2 illustrates example input-first mode and output-first mode configurations of an input scheduler for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 12-1 illustrates a first example implementation of an input scheduler for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 12-2 illustrates a second example implementation of an input scheduler for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 13 illustrates an example implementation of an output scheduler for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 14 illustrates a third example implementation of a decoupling circuit for implementing aspects of scheduling using decoupled grant and accept stages;FIG. 15 illustrates a first example method for performing aspects of scheduling using decoupled grant and accept stages;FIG. 16 illustrates a second example method for performing aspects of scheduling using decoupled grant and accept stages; andFIG. 17 illustrates an example computing system embodying, or in which techniques may be implemented that enable use of, scheduling using decoupled grant and accept stages.DETAILED DESCRIPTION

[0009] Technological advancements enable a system-on-chip (SoC) to be designed with a larger quantity of subsystems to further expand feature capabilities of an electronic device. The larger quantity of subsystems, however, can increase a complexity' for establishing communications between these subsystems. To address this issue, a system-on-chip can be implemented with a network-on-chip (NOC), which provides an interface for the subsystems to communicate with each other. An architecture (or topology) of the network-on-chip can significantly impact factors of the system-on-chip, including latency, throughput, and scalability.

[0010] Some network-on-chips have a mesh-topology’, which can be readily scaled and are relatively easy to implement. The mesh topology, however, can degrade latency performance and can have a higher risk for bottlenecks. Other network-on-chips can utilize a crossbar topology. The crossbar topology can have better performance in terms of latency and throughput compared to the mesh topology . This performance improvement, however, can come at the cost of additional complexity and a larger footprint. In some implementations, the crossbar topology is not as readily scalable as the mesh topology.

[0011] One type of crossbar topology7can improve throughput by performing multiple iterations of scheduling arbitration. This iterative approach, however, is challenging to scale. As a quantity of ports and virtual channels increase, the complexity of the scheduling arbitration cansignificantly increase and can result in additional iterations. A larger quantity of iterations can lead to increased latency and limit frequencies at which subsystems of the system-on-chip can communicate. In many cases, use of this iterative approach is limited to smaller system-on-chips (e.g., system-on-chips with fewer subsystems) and / or is limited to system-on-chips with less stringent frequency requirements (e.g., to system-on-chips that operate at lower clock frequencies). As seen in the examples above, existing on-chip networking techniques involve some sort of the tradeoff betw een latency, throughput, and scalability.

[0012] To address this challenge, techniques are described for scheduling using decoupled grant and accept stages. In example aspects, an arbiter of a crossbar scheduler uses buffering to decouple the grant and accept stages for scheduling. The buffering divides the scheduling operation into tw o operations, which can be performed in parallel. A first operation involves input scheduling, which transfers data from an initiator to a decoupling circuit of the arbiter as part of the grant stage. A second operation involves output scheduling, which transfers data from the decoupling circuit to a target as part of the accept stage. With the decoupling circuit, data flows in a pipeline fashion from the initiator to the target across at least two cycles. The decoupling of the grant and accept stages provides additional flexibility in performing the input scheduling using an input-first mode or an output-first mode. By decoupling the grant and accept stages, the arbiter can have a scalable architecture that supports high-speed and high-throughput scheduling.Operating Environment

[0013] FIG. 1 is an illustration of an example environment 100 in which scheduling using decoupled grant and accept stages can be implemented. In the example environment 100, a computing device 102 provides features and / or services for a user 104. Although depicted as a smartphone, the computing device 102 can include other types of devices, including those described with respect to FIG. 2. The computing device 102 includes at least one system-on-chip (SOC) 106 (SOC 106). The system-on-chip can be implemented with electronic circuitry, a microprocessor, memory, input-output (I / O) control logic, communication interfaces, firmware, and / or software useful to provide functionalities of the computing device 102.

[0014] The system-on-chip 106 includes multiple subsystems 108-1, 108-2... 108-S, where S represents a positive integer. The subsystems 108 can also be referred to as agents, modules, intellectual-property blocks (IP blocks), intellectual-property cores, or virtual components. Example subsystems 108 can include a central processing unit (CPU), a graphics processing unit (GPU), an image processing unit (IPU), a modem, a digital signal processor (DSP), a neural processing unit (NPU), a processor, a memory. a sensor, an analog circuit, a digital circuit,components that handle application-specific processing functions, and so forth. To perform one or more functions of the computing device 102, at least one of the subsystems 108 can communicate with at least another one of the subsystems 108. In some situations, two or more subsystems 108 can communicate with two or more other subsystems 108 during a same time period.

[0015] The system-on-chip 106 also includes at least one network-on-chip (NOC) 1 10 (NOC 110) to provide a communication interface between the subsystems 108. In some implementations, the system-on-chip 106 includes multiple network-on-chips 110, which connect different sets of subsystems 108 together. The network-on-chip 110 can also be considered another subsystem 108 of the system-on-chip 106.

[0016] In example implementations, the network-on-chip 110 is implemented using a crossbar scheduler 112. The crossbar scheduler 112 performs aspects of scheduling using decoupled grant and accept stages, as further described below. With a topology that supports the decoupling of the grant and accept stages, the crossbar scheduler 112 can have a scalable architecture 1 14 capable of performing high-speed, high-throughput scheduling 116. Using the described techniques, the crossbar scheduler 112 can be designed in a manner that avoids the typical tradeoffs between latency, throughput, and scalability, which can impact the design and performance of other types of crossbar schedulers. The crossbar scheduler 112 can also realize the high-speed, high-throughput scheduling 116 without performing multiple iterations of arbitration.

[0017] The scalable architecture 114 enables the crossbar scheduler 112 to be readily modified for supporting communications between various quantities of subsystems 108 without substantially increasing a size or cost of the crossbar scheduler 112. The high-speed, high- throughput scheduling 116 means that the crossbar scheduler 112 can continue to realize target levels of performance in terms of latency and throughput while facilitating communication across larger quantities of subsystems 108. In example implementations, the high-speed, high- throughput scheduling 116 can enable the network-on-chip 106 to perform operations based on a clock signal having a frequency that is greater than 400 megahertz (MHz), including frequencies above 1 or 2 gigahertz (GHz). Example frequencies of the clock signal can be approximately equal to 500 MHz, 800 MHz, 1 GHz, 1.4 GHz, 2 GHz, and so forth. While the techniques for scheduling using decoupled grant and accept stages enables the network-on-chip 106 to operate a higher frequencies relative to other network-on-chips, the network-on-chip 106 is not limited to these higher frequencies and can perform operations based on a clock signal having a frequency that is less than 400 MHz.

[0018] Although the techniques for scheduling using decoupled grant and accept stages are generally described with respect to the crossbar scheduler 112, these techniques can also be applied to other ty pes of schedulers. The techniques for scheduling using decoupled grant and accept stages can be particularly beneficial for schedulers that support communications between multiple initiators and multiple targets. The components of the system-on-chip 106 (e.g., the subsystems 108 and the network-on-chip 1 10) may alternatively be implemented within other types of integrated circuits or embedded systems, such as a microchip, an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a digital signal processor (DSP), a programmable system-on-chip (PSoC). system-in-package (SiP), controller, and so forth. The computing device 102 is further described with respect to FIG. 2.

[0019] FIG. 2 illustrates an example computing device 102. The computing device 102 is illustrated with various non-limiting example devices including a desktop computer 102-1, a tablet 102-2, a laptop 102-3, a television 102-4, a computing watch 102-5, computing glasses 102-6, a gaming system 102-7, a router 102-8, and a vehicle 102-9. Other devices may also be used, such as a home service device, a smart speaker, a smart thermostat, a baby monitor, a Wi-Fi™ router, a drone, a trackpad, a drawing pad, a netbook, an e-reader, a home automation and control system, a wall display, and another home appliance. Note that the computing device 102 can be wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances).

[0020] The computing device 102 includes at least one system-on-chip 106. The system-on- chip 106 includes the subsystems 108-1 to 108-S and the network-on-chip 110. In example implementations, the network-on-chip 110 is implemented using the crossbar scheduler 112. The crossbar scheduler 112 is implemented using at least one arbiter 202, which provides contention- free matching between input-output pairs of the subsystems 108. The arbiter 202 performs aspects of scheduling and can handle situations in which two or more subsystems 108 request communications with a same subsystem 108. In general, the arbiter 202 represents any type of scheduler that is used to implement the network-on-chip 110 and has a topology that enables decoupling of the grant and accept stages.

[0021] To implement the scalable architecture 114 and to perform the high-speed, high- throughput scheduling 116 of FIG. 1, the arbiter 202 includes at least one input scheduler 204, at least one output scheduler 206, and at least one decoupling circuit 208. The decoupling circuit 208 is coupled between the input scheduler 204 and the output scheduler 206, as shown in FIGs. 3 and 4. The input scheduler 204 performs one aspect of scheduling at a grant stage, as further described with respect to FIG. 3. Example implementations of the input scheduler 204 are further describedwith respect to FIG. 10 to 12. The output scheduler 206 performs another aspect of the scheduling at an accept stage, as further described with respect to FIG. 3. An example implementation of the output scheduler 206 is further described with respect to FIG. 13.

[0022] The decoupling circuit 208 provides buffering to decouple the grant and accept stages. This buffering divides the scheduling operation into two operations: input scheduling and output scheduling, both of which are further described with respect to FIG. 3. Based on the decoupling provided via the decoupling circuit 208, the input scheduler 204 can execute the grant stage and allow data to progress downstream without waiting for input from the output scheduler 206. Also, the output scheduler 206 can execute the accept stage and allow data to progress downstream without waiting for input from the input scheduler 204. The decoupling circuit 208 can also alleviate head-of-line blocking (HOL) and support different traffic conditions. Generally speaking, the decoupling circuit 208 enables the crossbar scheduler 112 to realize a higher level of throughput without relying on an iterative scheduling approach and while supporting multiple layers of arbitration (e.g., arbitration based on traffic classes (TCs), virtual channels (VCs), quality -of-service (QoS), and / or ports).

[0023] Although the decoupling circuit 208 can increase an overall footprint of the arbiter 202, the decoupling circuit 208 enables the arbiter 202 to have the scalable architecture 114. For situations in which the arbiter 202 supports communications between a relatively large quantity of subsystems 108, this scalable architecture 1 14 can enable the arbiter 202 to have a smaller footprint and / or have less complexity compared to other types of arbiters. Various implementations of the decoupling circuit 208 can support higher levels of throughput or can have a smaller impact on a footprint of the arbiter 202. Example implementations of the decoupling circuit 208 are further described with respect to FIGs. 8, 9, and 14.

[0024] The computing device 102 can also include a netw ork interface 210 for communicating data over wired, wireless, or optical networks. For example, the network interface 210 may communicate data over a local-area-network (LAN), a wireless local-area-network (WLAN), a personal-area-network (PAN), a wire-area-network (WAN), an intranet, the Internet, a peer-to- peer network, point-to-point network, a mesh network, Bluetooth™, and the like. The computing device 102 may also include a display 212. A relationship between the arbiter 202 and the subsystems 108 is further described with respect to FIG. 3.Scheduling using Decoupled Grant and Accept Stages

[0025] FIG. 3 illustrates an example operation of the arbiter 202. In the depicted configuration, the arbiter 202 is coupled between initiators 302-1, 302-2... 302-N and targets 304-1, 304-2...304-M, where N and M are positive integers that may or may not be equal to each other. Each initiator 302 represents one of the subsystems 108 capable of transmitting data. Each target 304 represents another one of the subsystems 108 capable of receiving data.

[0026] The arbiter 202 includes input ports 306-1, 306-2... 306-N and output ports 308-1, 308-2... 308-M. The input ports 306-1 to 306-N are coupled to corresponding initiators 302-1 to 302-N. Likewise, the output ports 308-1 to 308-M are coupled to corresponding targets 304-1 to 304-M. Communications between the initiators 302 and the targets 304 pass through the corresponding input ports 306 and the corresponding output ports 308. The input ports 306 can also be referred to as ingress ports, and the output ports 308 can also be referred to as egress ports.

[0027] During operation, the arbiter 202 pairs one or more of the input ports 306 to one or more of the output ports 308 to enable data to be transferred from one or more initiators 302 to one or more targets 304. To schedule this data transfer, the arbiter 202 performs two parallel operations, which are represented by input scheduling 310 and output scheduling 312.

[0028] Input scheduling 310 involves transferring data from one or more initiators 302 to the decoupling circuit 208. The input scheduling 310 implements a grant stage 314, which represents a first main stage (e. g. , a beginning stage) of the arbiter 202. The grant stage 314 involves granting requests. In more detail, the grant stage 314 performs arbitration to determine which requests to qualify. The grant stage 314 can arbitrate multiple requests from an initiator 302 and can select one of the requests as a candidate for the arbitration performed at the target 304 as part of the accept stage 318. In some implementations of the arbiter 202, the input scheduling 310 also implements an auxiliary accept stage 316 (aux. accept stage 316), as further described with respect to FIG. 11-1. In general, the grant stage 314 and the auxiliary accept stage 316 are executed in series during a same cycle (e.g., a same clock cycle).

[0029] Output scheduling 312 involves transferring data from the decoupling circuit 208 to one or more targets 304. The output scheduling 312 implements an accept stage 318, which represents a second main stage (e.g., a last stage) of the arbiter 202. The accept stage 318 sends data to the targets 304. In more detail, the accept stage 316 performs arbitration to determine which data to send to a target 304. The accept stage 318 represents a final arbitration that is performed at each target 304.

[0030] The arbiter 202 uses buffering to cause the grant stage 314 and the accept stage 318 to be decoupled 320 from each other. Due to the decoupling, the grant stage 314 and the accept stage 318 can be executed in parallel. This distinguishes the arbiter 202 from other types of arbiters that operate using a same-cycle grant-accept sequence.

[0031] With the decoupling of the grant stage 314 and the accept stage 318, the arbiter 202 can transfer data in a pipeline fashion from the initiators 302 to the targets 304 using local pointers, local credit loops, and local port locks. This pipeline-like propagation of the data is further described with respect to FIG. 7. The decoupling of the grant stage 314 and the accept stage 318 also allows for better placement and span of control and data routes within the architecture of the arbiter 202, which enables the arbiter 202 to have the scalable architecture 1 14.

[0032] The decoupling of the grant stage 314 and the accept stage 318 further provides flexibility in implementing the input scheduling 310 using an input-first mode 322 or an output-first mode 324. Both modes 322 and 324 support a many-to-many parallel matching process. Each mode 322 and 324 can have its own advantages and disadvantages depending on the quantity of initiators 302 (e.g., N), the quantity of targets 304 (e.g., AT), and the quantity of requests that are received by the arbiter 202 at a given time. The input-first mode 322 and the output-first mode 324 are further described with respect to FIGs. 10 to 12.

[0033] During the grant stage 314, the initiators 302 can send requests 326 to the arbiter 202. Each request 326 can be associated with a particular target 304 and a particular virtual channel. Each initiator 302 can send multiple requests 326, as further described with respect to FIG. 5. The arbiter 202 sends grants 328 to the initiators 302. The arbiter 202 also receives data 330 from the initiators 302 based on the grants 328. The data 330 can also be referred to as upstream data.

[0034] The data 330 stored by an initiator 302 can include one or more packets. Each packet can include one or more beats. A beat represents a portion of a packet that can be transferred from the initiator 302 to the arbiter 202 during a single cycle. An example beat is further described with respect to FIG. 7.

[0035] During the accept stage 318, each target 304 provides its credit availability 332 to the arbiter 202. This can be done without a target 304 having knowledge of the credit availability7332 of the other targets 304. The arbiter 202 also sends credit control signals 334 (credit control sig. 334) to the targets 304. The credit control signals 334 can cause the targets 304 to decrease their available credit. The targets 304 receive data 336 from the arbiter 202. The data 336 can also be referred to as downstream data. Generally speaking, the data 330 transmitted by one of the initiators 302-1 to 302-N is received as the data 336 by one of the targets 304-1 to 304-M. Although described with respect to pairing an initiator 302 and a target 304, each stage (e.g., the grant stage 314 and the accept stage 318) can also be performed based on a traffic class, a virtual channel, and / or a quality-of-service. The input scheduling 310 (e.g., the grant stage 314) and the output scheduling 312 (e.g., the accept stage 318) are performed using the input scheduler 204 and the output scheduler 206, respectively, as further described with respect to FIG. 4.

[0036] FIG. 4 illustrates an example relationship between the initiators 302, the targets 304, and the arbiter 202. In the depicted configuration, the arbiter 202 includes multiple input schedulers 204-1, 204-2... 204-N, multiple output schedulers 206-1, 206-2... 206-M, and the decoupling circuit 208. The input schedulers 204-1 to 204-N are respectively coupled to the input ports 306-1 to 306-N. The initiators 302 and the input schedulers 204 communicate the requests 326 and the grants 328 via the input ports 306. The output schedulers 206-1 to 206-M are respectively coupled to the output ports 308-1 to 308-M. The targets 304 and the output schedulers 206 communicate the credit availability 332 and the credit control signals 334 via the output ports 308.

[0037] The decoupling circuit 208 is coupled between the input schedulers 204 and the output schedulers 206. The decoupling circuit 208 is also coupled to the initiators 302 and the target 304 via the input ports 306 and the output ports 308, respectively. Through the input ports 306, the decoupling circuit 208 receives the data 330 from the initiators 302. For example, the decoupling circuit 208 can receive data 330-1, 330-2... 330-N from the initiators 302-1, 302-2... 302-N, respectively. Through the output ports 308, the decoupling circuit 208 sends the data 336 to the targets 304. For example, the decoupling circuit 208 can send data 336-1, 336-2. . . 336-M to the targets 304-1, 304-2... 304-M, respectively.

[0038] The decoupling circuit 208 includes multiple skid buffers 402. The skid buffers 402 enable data to be written to and read from the decoupling circuit 208 during a single cycle. In this way, the skid buffers 402 support the decoupling of the grant stage 314 and the accept stage 318 by enabling a first beat to be read from the decoupling circuit 208 and by enabling a second beat to be written to the decoupling circuit 208 during a same time interval, as further described with respect to FIG. 7.

[0039] The decoupling circuit 208 can have a virtual-input-queue (VIQ) structure 404 (VIQ structure 404) or a virtual-channel-queue (VCQ) structure 406 (VCQ structure 406). Example implementations of the decoupling circuit 208 having the virtual-input-queue structure 404 and the virtual-channel-queue structure 406 are further described with respect to FIGs. 8 and 9, respectively. With the virtual-input-queue structure 404, the input schedulers 204 can use the input-first mode 322 to perform the input scheduling 310. The virtual -input-queue structure 404 enables parallel buffering to be performed per initiator 302, which can improve throughput.

[0040] The virtual-channel-queue structure 406 can have a smaller footprint compared to the virtual-input-queue structure 404, which can be beneficial for incorporating the system-on- chip 106 within space-constrained devices. This smaller footprint comes at the cost of decreased throughput performance compared to an implementation using the virtual-input-queuestructure 404. With the virtual-channel-queue structure 406, the input schedulers 204 can use the input-first mode 322 or the output-first mode 324 to perform the input scheduling 310.

[0041] During operation, the input schedulers 204 perform input scheduling 310 as part of the grant stage 314. The input scheduling 310 enables the data 330 to make forward progress towards the targets 304. The decoupling circuit 208 temporarily stores the data 330 to decouple the grant stage 314 and the accept stage 318. The output schedulers 206 perform output scheduling 312 as part of the accept stage 318. The output scheduling 312 enables the data 336 to make forward process from the decoupling circuit 208 towards the targets 304.

[0042] Due to the decoupling of the grant stage 314 and the accept stage 318, the input scheduling 310 and the output scheduling 312 can be performed independent of one another. In other words, the grant stage 314 and the accept stage 318 can be executed without implementing a loop-back feature (e.g., without performing multiple scheduling iterations). This means that each input scheduler 204 perfbnns input scheduling 310 at a corresponding initiator 302 independent of the output scheduling 312 performed at the targets 304. Likewise, each output scheduler 206 performs output scheduling 312 at a corresponding target 304 independent of the input scheduling 310 performed at the initiators 302. In general, the input schedulers 204 and the output schedulers 206 can operate in parallel on a given cycle. During each cycle, the input schedulers 204 and the output schedulers 206 can make scheduling decisions (e.g., perform arbitration) to move data forward. Example operations of the input schedulers 204 and the output schedulers 206 are further described with respect to FIG. 5.

[0043] FIG. 5 illustrates example internal and external communications involving the arbiter 202 for performing scheduling using decoupled grant and accept stages. The internal communications occur between the input scheduler 204, the output scheduler 206, and the decoupling circuit 208. The external communications occur between the arbiter 202, the initiator 302, and the target 304.

[0044] For simplicity7, the external communications in FIG. 5 are shown between the arbiter 202 and one of the initiators 302-1 to 302-N as well as between the arbiter 202 and one of the targets 304-1 to 304-M. The example external communications shown in FIG. 5 are similar for the remaining initiators 302 and targets 304 not explicitly shown. Also, the internal communications in FIG. 5 are shown with respect to one of the input schedulers 204-1 to 204-N and one of the output schedulers 206-1 to 206-M. The example internal communications shown in FIG. 5 are similar for the remaining input schedulers 204 and output schedulers 206 not explicitly shown.

[0045] In the depicted configuration, the initiator 302 includes control queues 502, such as queue 502-1 to 502-M. Each queue 502-1 to 502 -M corresponds to one of the targets 304-1 to304-M. A queue 502 can have multiple buffers 504 (e.g., slots or entries), such as buffers 504-1 to 504-V, where V represents a positive integer. Each buffer 504 can correspond to a virtual channel of the initiator 302. In this case, the initiator 302 supports a quantity of virtual channels represented by the variable V.

[0046] The initiator 302 also includes at least one memory 506 to store the data 330. The memory 506 can represent any type of computer-readable storage medium. In a first example implementation, the memory 506 is implemented using volatile memory, such as a cache memory, a random-access memory', or a portion of a memory array. In a second example implementation, the memory 506 is implemented using non-volatile memory, such as a fuse array, a flash memory, metal bits, a programmable read-only memory, or a one-time programmable memory. In a third example implementation, the memoiy 506 can include a set of registers.

[0047] The initiator 302 can send one or more requests 326 to the input scheduler 204. Each request 326 corresponds to one of the buffers 504. For example, requests 326-1 to 326-V respectively correspond to the buffers 504-1 to 504-V. Other requests 326 correspond to the buffers 504 within the queue 502-M. In general, each buffer 504 asserts a request 326 if the buffer 504 is not empty (e.g., stores data). A buffer 504 does not assert a request 326 if it is empty (e.g., does not store data). The initiator 302 can send a total quantity of requests 326 that is equal to a product of V and M. In some implementations, each buffer 504 is associated with a different classification and / or a different priority.

[0048] The decoupling circuit 208 sends internal credit availability 508 to the input scheduler 204. The internal credit availability 508 can also be referred to as a local credit availability. In some implementations, the internal credit availability 508 represents a multi-bit signal, with each bit indicating whether a corresponding skid buffer 402 is not empty or empty. In general, the internal credit availability 508 indicates which skid buffers 402 are available for storing the data 330. In some implementations, the internal credit availability 508 can specify the credit availability corresponding to each target 304 and for each virtual channel. In other implementations, the internal credit availability 508 can also specify the credit availability with respect to each initiator 302. The internal credit availability 508 enables the input scheduler 204 to monitor the availability of the decoupling circuit 208 to qualify the requests 326.

[0049] The input scheduler 204 determines which requests 326 to grant based on the internal credit availability 508. In other words, the input scheduler 204 arbitrates which data 330 is to be forwarded to the decoupling circuit 208. The input scheduler 204 sends the grant 328 based on the determination. The grant 328 identifies which request 326 is granted and causes the initiator 302 to send the data 330 associated with the request 326.

[0050] The decoupling circuit 208 receives the data 330 from the initiator 302 and stores the data 330. The decoupling circuit 208 sends internal requests 510 to the output scheduler 206. The internal requests 510 can also be referred to as local requests. Each internal request 510 corresponds to one of the skid buffers 402 that store the data 330. Skid buffers 402 that are empty (e.g., do not store data 330) do not send an internal request 510. By sending at least one internal request 510, the decoupling circuit 208 initiates an arbitration cycle associated with at least one of the targets 304. This arbitration cycle is performed by the output scheduler 206 at the accept stage 318 through output scheduling 312.

[0051] The output scheduler 206 receives credit availability 332 from the target 304. The credit availability 332 enables the output scheduler 206 to monitor the availability of the target 304 to quality the internal requests 510. Based on the credit availability 332, the output scheduler 206 determines which internal request 510 to grant. In some cases, the determination is based on a priority associated with the internal request 510. The priority can be determined based on the traffic class, the virtual channel, the quality-of-service, and / or the port associated with the data 330 that is stored within the decoupling circuit 208. In other words, the output scheduler 206 arbitrates which data 336 is to be sent to the target 304. The output scheduler 206 also sends the credit control signal 334 to the target 304. The credit control signal 334 can cause the target 304 to decrease its credit availability 332.

[0052] The output scheduler 206 sends an internal grant 512 to the decoupling circuit 208. The internal grant 512 can also be referred to as a local grant. The internal grant 512 identifies which internal request 510 is granted and causes the decoupling circuit 208 to send the data 336 associated with the internal request 510 to the target 304. In this manner, the arbiter 202 transfers data from the initiator 302 to the target 304.

[0053] In the example depicted in FIG. 5, example external communications between the initiator 302 and the arbiter 202 include the one or more requests 326 sent by the initiator 302 to the arbiter 202, the one or more grants 328 sent by the arbiter 202 to the initiator 302, and the data 330 sent by the initiator 302 to the arbiter 202. Example external communications between the target 304 and the arbiter 202 include the credit availability 332 sent by the target 304 to the arbiter 202, the credit control signal 334 sent by the arbiter 202 to the target 304, and the data 336 sent by the arbiter 202 to the target 304. Example internal communications include the internal credit availability 508 sent from the decoupling circuit 208 to the input scheduler 204, the internal requests 510 sent from the decoupling circuit 208 to the output scheduler 206, and the internal grant 512 sent from the output scheduler 206 to the decoupling circuit 208. The internal andexternal communications performed using the arbiter 202 are further described with respect to FIG. 6.

[0054] FIG. 6 illustrates example transactions between the initiator 302, the arbiter 202, and the target 304. At 602, the initiator 302 sends one or more requests 326 to the input scheduler 204 of the arbiter 202. At 604, the decoupling circuit 208 sends the internal credit availability 508 (int. credit avail. 508) to the input scheduler 204. Although the step at 604 is shown to occur after 602, other situations are possible in which the step at 604 is performed prior to the step at 602.

[0055] At 606, the input scheduler 204 sends the grant 328 to the initiator 302. At 608, the initiator 302 sends the data 330 to the decoupling circuit 208. The decoupling circuit 208 stores the data 330 at 608.

[0056] At 610, the decoupling circuit 208 sends internal requests 510 to the output scheduler 206. At 612, the target 304 sends its credit availability 332 to the output scheduler 206. Although the step at 612 is shown to occur after 610, other situations are also possible in which the step at 612 is performed prior to the step at 610.

[0057] At 614, the output scheduler 206 sends the internal grant 512 to the decoupling circuit 208. The output scheduler 206 also sends the credit control signal 334 (credit cntrl sig. 334) to the target 304 at 616. Although steps 614 and 616 are shown as separate steps in FIG. 6, in some cases these steps can be performed together or can be considered to be performed as part of a single step. Also, in various implementations, the step 616 can be performed prior to the step 614 or after the step 614 (as shown in FIG. 6). At 618, the decoupling circuit 208 sends the data 336 to the target 304.

[0058] The steps described at 602, 604, 606. and 608 are associated with the grant stage 314. These steps are performed during a first cycle 620-1 (e.g., during a first clock cycle or during a first time period). The first cycle 620-1 represents a single clock cycle. The steps described at 610, 612, 614, 616, and 618 are associated with the accept stage 318 and are performed during a second cycle 620-2 (e g., during a second clock cycle or during a second time period). The second cycle 620-2 also represents a single clock cycle. The second cycle 620-2 represents a next clock cycle that occurs after the first cycle 620-1. This causes the data 330 and 336 to progress from the initiator 302 to the target 304 in a pipeline fashion, as further described with respect to FIG. 7.

[0059] FIG. 7 illustrates an impact of decoupling the grant stage 314 and the accept stage 318 on the propagation of data. At 700, data is shown to propagate, over time, from the initiator 302, through the decoupling circuit 208, and to the target 304 in a pipeline fashion. The data isrepresented by multiple beats 702, including beats 702-1, 702-2, and 702-3. The beats 702 can be associated with a same packet, different packets, or some combination thereof.

[0060] During a first time interval 704-1 and as part of the grant stage 314, the arbiter 202 causes the first beat 702-1 to be read from the initiator 302 and written to the decoupling circuit 208. During a second time interval 704-2 and as part of the accept stage 318, the arbiter 202 causes the first beat 702-1 to be read from the decoupling circuit 208 and sent to the target 304. With respect to the first beat 702-1, the first time interval 704-1 represents the first cycle 620-1 and the second time interval 704-2 represents the second cycle 620-2 of FIG. 6.

[0061] During the second time interval 704-2, the arbiter 202 also causes the second beat 702-2 to be read from the initiator 302 and written to the decoupling circuit 208 as part of the grant stage 314. During a third time interval 704-3 and as part of the accept stage 318, the arbiter 202 causes the second beat 702-2 to be read from the decoupling circuit 208 and sent to the target 304. With respect to the second beat 702-2, the second time interval 704-2 represents the first cycle 620-1 and the third time interval 704-3 represents the second cycle 620-2 of FIG. 6.

[0062] During the third time interval 704-3, the arbiter 202 also causes the third beat 702-3 to be read from the initiator 302 and written to the decoupling circuit 208. With respect to the third beat 702-3, the third time interval 704-3 represents the first cycle 620-1. In general, the time intervals 704-1, 704-2. and 704-3 represent different cycles of a clock signal used for scheduling. As shown in FIG. 7, the grant stage 314 and the accept stage 318 can operate on different beats 702 during a same time interval 704 due to the decoupled 320 nature of the grant stage 314 and the accept stage 318. Example implementations of the decoupling circuit 208 are further described with respect to FIGs. 8 and 9.

[0063] FIG. 8 illustrates a first example implementation of the decoupling circuit 208, which has the virtual -input-queue structure 404. This means that the skid buffers 402 are per virtual channel and per initiator 302, as further described below. The virtual-input-queue structure 404 ensures each initiator 302 can receive a grant 328 if the decoupling circuit 208 has availability in any virtual channel designated for that initiator 302.

[0064] The decoupling circuit 208 includes per-target set of buffers 802-1 to 802-M. The pertarget set of buffers 802-1 to 802-M respectively correspond to the targets 304-1 to 304-M. Within each per-target set of buffers 802, the decoupling circuit 208 includes virtual-input queues 804-1 to 804-N. The virtual -input queues 804-1 to 804-N respectively correspond to the initiators 302-1 to 302-N.

[0065] Within each virtual-input queue 804, the decoupling circuit 208 includes multiple skid buffers 402. Each skid buffer 402 corresponds to a virtual channel 806. For example, the skidbuffers 402-1 to 402-V respectively correspond to the virtual channels 806-1 to 806-V. In this example, the quantity of skid buffers 402 is based on the quantity of virtual channels (e.g., F), the quantity7of initiators (e.g., N), and the quantity7of targets (e.g., M). As such, each skid buffer 404 has an index associated with a particular target 304, a particular initiator 302, and a particular virtual channel 806. In this manner, each skid buffer 402 corresponds to a different virtual channel 806 of a different initiator 302.

[0066] The virtual-input-queue structure 404 enables the arbiter 202 to improve its throughput performance. However, depending on the quantity of initiators 302 (e.g., A), a size and cost of the decoupling circuit 208 with the virtual-input-queue structure 404 may be too large for some implementations. To decrease the size and / or the cost of the decoupling circuit 208, the decoupling circuit 208 can instead be implemented using the virtual-channel-queue structure 406, as further described with respect to FIG. 9.

[0067] FIG. 9 illustrates a second example implementation of the decoupling circuit 208, which has the virtual-channel-queue structure 406. This means that skid buffers 402 within the decoupling circuit 208 are per virtual channel 806, as further described below.

[0068] The decoupling circuit 208 includes per-target set of buffers 802-1 to 802-M. The pertarget set of buffers 802-1 to 802-M respectively correspond to the targets 304-1 to 304-M. Each per-target set of buffers 802 includes skid buffers 402-1 to 402-V. Each skid buffer 402 corresponds to a virtual channel 806. For example, the skid buffers 402-1 to 402-V respectively correspond to the virtual channels 806-1 to 806-V. In this example, the quantity of skid buffers 402 is based on the quantity of virtual channels (e.g., F) and the quantity of targets (e.g., M). As such, each skid buffer 806 has an index associated with a particular target 304 and a particular virtual channel 806. In this case, the virtual channels 806 are common or shared by the initiators 302.

[0069] As can be seen in FIGs. 8 and 9, a size of the decoupling circuit 208 of FIG. 9 is less than a size of the decoupling circuit 208 of FIG. 8. In other words, the decoupling circuit 208 of FIG. 9 has fewer skid buffers 402 compared to the decoupling circuit 208 of FIG. 8. This reduction in the quantity of skid buffers 402 comes at the cost of increased complexity at the input scheduler 204, a possible increase in latency performance, and a potential decrease in throughput performance for implementations that use a single virtual channel. In most cases, the possible increase in latency performance can be relatively small and does not significantly degrade performance of the network-on-chip 106.

[0070] To utilize the decoupling circuit 208 of FIG. 9, the input scheduler 204 performs the auxiliary7accept stage 316, as further described in FIG. 11-1. The auxiliary7accept stage 316provides the additional arbitration that is necessary to resolve conflicts associated with different initiators 302 requesting the send data to a same target 304 using a same virtual channel 806. Example implementations of the arbiter 202 utilizing the decoupling circuits 208 of FIGs. 8 and 9 are further described with respect to FIGs. 10 and 11-1, respectively.

[0071] FIG. 10 illustrates a first example implementation of the arbiter 202 for implementing aspects of scheduling using decoupled grant and accept stages. In this example, the decoupling circuit 208 has the virtual-input-queue structure 404. The virtual-input-queue structure 404 enables the data 330 from multiple initiators 302 to be stored in the per-target set of buffers 802 corresponding to a same target 304. In this way, throughput performance can be improved at the cost of a larger quantity of skid buffers 402.

[0072] The input scheduler 204 executes the grant stage 314, and the output scheduler 206 executes the accept stage 318. The input scheduler 204 can perform input scheduling 310 using the input-first mode 322. Generally speaking, input scheduling 310 is performed by each input scheduler 204 at each initiator 302. The input scheduling 310 allows data 330 to make forward progress to the per-virtual-channel, per-initiator skid buffers 402 within the virtual-input-queue structure 404 of the decoupling circuit 208. This input scheduling 310 can be performed without relying on feedback from the output scheduler 206.

[0073] If any credit is available in the decoupling circuit 208 and the initiators 302 have qualified requests 326 for a target 304, the arbiter 202 can guarantee to send a grant 328 and store the data 330 in the decoupling circuit 208. With the virtual -input-queue structure 404, matching can be maximized so long as there is available space within the decoupling circuit 208. This means that matching can be maximized during a single grant stage 314 and without perfonning multiple iterations.

[0074] The virtual-input-queue structure 404 of the decoupling circuit 208 also enables matching to be maximized during a single accept stage 318 and without performing multiple iterations. As such, the data 336 can be sent to a target 304 if the target 304 has at least one credit available. Example implementations of the input scheduler 204 and the output scheduler 206 are further described with respect to FIGs. 12 and 13.

[0075] Although the virtual-input-queue structure 404 can be associated with high throughput, it can increase a size of the decoupling circuit 208. For situations in which it is desirable to design the decoupling circuit 208 with a smaller footprint, the decoupling circuit 208 can alternatively be implemented with the virtual-channel-queue structure 406, as further described with respect to FIG. 11-1.

[0076] FIG. 11-1 illustrates a second example implementation of the arbiter 202 for implementing aspects of scheduling using decoupled grant and accept stages. In this example, the decoupling circuit 208 has the virtual-channel-queue structure 406. As such, the decoupling circuit 208 does not have per-initiator skid buffers 402 as in the virtual-input-queue structure 404 shown in FIG. 10. This means that a single initiator 302 can send data 330 to a skid buffer 402 using one of the virtual channels 806. A conflict will otherwise arise, however, if multiple initiators 302 send requests 326 to transfer data 330 to a same target 304 using a same virtual channel 806.

[0077] To address this potential conflict, the input scheduler 204 performs the auxiliary accept stage 316 in addition to the grant stage 314. With the auxiliary accept stage 316. the input scheduler 204 grants one of the initiators 302 access to the target 304 using the virtual channel 806. The occurrence of this potential conflict can reduce the throughput performance of the arbiter 202 of FIG. 11-1 relative to the arbiter 202 of FIG. 10. However, this reduction in performance is limited to situations in which two or more initiators 302 are attempting to use the same virtual channel 806. In other situations in which multiple initiators are using different virtual channels 806, there is no conflict and no performance degradation. The throughput performance of the arbiter 202 of FIG. 11-1 can be similar to a crossbar scheduler that performs scheduling using a single iteration. The virtual-channel-queue structure 406 enables the arbiter 202 of FIG. 11-1 to have a smaller footprint than the arbiter 202 of FIG. 10.

[0078] In this example, the input scheduler 204 executes the grant stage 314 and the auxiliary accept stage 316 in series during a same cycle 620. The virtual-channel-queue structure 406 provides the input scheduler 204 flexibility in using the input-first mode 322 or the output-first mode 324, as further described with respect to FIGs. 12-1 and 12-2. In the input-first mode 322, the auxiliary accept stage 316 is performed on a per- virtual-channel basis. In the output-first mode 324, the grant stage 314 is performed on a per-virtual-channel basis. The output scheduler 206 executes the accept stage 318, which is performed in parallel to the series combination of the grant stage 314 and the auxiliary accept stage 316. Different operations of the input scheduler 204 with respect to the input-first mode 322 and the output-first mode 324 are further described with respect to FIG. 11-2.

[0079] FIG. 11-2 illustrates example input-first mode 322 and output-first mode 324 configurations of the input scheduler 204 at 1100-1 and 1100-2, respectively. In both the input- first mode 322 and the output-first mode 324, the grant stage 314 and the auxiliary accept stage 316 are performed in senes. Also, the input scheduler 204 performs the grant stage 314 prior to performing the auxiliary accept stage 316.

[0080] In the input-first mode 322 at 1100-1, the grant stage 314 is performed on a full basis 1 102. This means that the grant stage 314 performs multi-layered arbitration. In an example implementation, the multiple layers can correspond to the traffic class, the virtual channels 806, the quality-of-service, and / or the ports. The auxiliary accept stage 316 is performed on a per- virtual -channel basis 1104. This means that the auxiliary accept stage 316 performs arbitration based on the virtual channels 806. Performing arbitration on the full basis 1102 differs from performing arbitration on the per-virtual-channel basis 1104 in that the full basis 1102 involves performing arbitration across a larger quantity of layers. As such, the grant stage 314 is more complex to execute compared to the auxiliary accept stage 316. The input scheduler 204 can include a multi-layered arbiter to implement the grant stage 314 and a per-virtual-channel port arbiter to implement the auxiliary accept stage 316, as further described with respect to FIG. 12-1.

[0081] In the output-first mode 324 at 1100-2, the grant stage 314 is performed on a per-virtual- channel basis 1104. This means that the grant stage 314 performs arbitration based on the virtual channels 806. The auxiliary accept stage 316 is performed on the full basis 1102. This means that the auxiliary accept stage 316 performs multi-layered arbitration. In an example implementation, the multiple layers can correspond to the traffic class, the virtual channels 806, the quality-of-service, and / or the ports. In contrast to the input-first mode 322, the auxiliary accept stage 316 is more complex to execute compared to the grant stage 314 in the output-first mode 324. The input scheduler 204 can include a per-virtual-channel port arbiter to implement the grant stage 314 and a multi-layered arbiter to implement the auxiliary accept stage 316, as further described with respect to FIG. 12-2. Example implementations of the input scheduler 204 are further described with respect to FIGs. 12-1 and 12-2.

[0082] FIG. 12-1 illustrates a first example implementation of the input scheduler 204. In various implementations, the input scheduler 204 can include an input interleaving controller 1202 and grant-stage arbiters 1204. The input interleaving controller 1202 provides input locking and request masking based on an input interleaving mode of the arbiter 202. If the input interleaving mode is enabled, the input interleaving controller 1202 enables an initiator 302 to selectively switch between sending beats 702 to different targets 304 across different cycles 602.

[0083] Consider an example in which the initiator 302 stores data 330 including a first packet that is to be sent to a first target 304-1 and a second packet that is to be sent to a second target 304-2. In an example scenario, the initiator 302 sends, during a first cycle 620-1, a first beat 702 of the first packet to the first target 304-1. During a second cycle, the initiator 302 sends a first beat 702 of the second packet to the second target 304-2. During a third cycle, the initiator 302 sends a second beat 702 of the first or second packet to the first or second target 304-1 and 304-2,respectively. In the manner described above, the initiator 302 interleaves transferring of the data 330 to the first target 304-1 and the second target 304-2. If the input interleaving mode is disabled, the input interleaving controller 1202 ensures the initiator 302 sends all beats 702 of the first packet to the first target 304-1 before sending beats 702 of the second packet to the second target 304-2.

[0084] The grant-stage arbiters 1204 provide arbitration for the grant stage 314. In some implementations, the input scheduler 204 can also include an output interleaving controller 1206 and / or auxiliary-accept-stage arbiters 1208. The output interleaving controller 1206 is coupled between the input interleaving controller 1202 and the grant-stage arbiters 1204. The grant-stage arbiters 1204 are coupled between the output interleaving controller 1206 and the auxiliary- accept-stage arbiters 1208. The coupling of the grant-stage arbiters 1204 to the auxiliary-accept- stage arbiters 1208 enables the grant stage 314 and the auxiliary accept stage 316 to be performed in series during a same c cle 620. The auxiliary-accept-stage arbiters 1208 provide arbitration for the auxiliary accept stage 316.

[0085] The output interleaving controller 1206 provides output locking and request masking based on an output interleaving mode of the arbiter 202. Depending on whether the output interleaving mode is enabled or disabled, a target 304 can switch to receiving different beats of the data 336 from different initiators 302 and from different virtual channels 806 or is limited to receiving a packet of the data 336 from a single initiator 302 prior to switching to receiving a packet of the data 336 from a different initiator 302.

[0086] If the input scheduler 204 is designed to support the input-first mode 322 and is coupled to a decoupling circuit 208 that has the virtual -input-queue structure 404 (as shown in FIG. 10), the input scheduler 204 may be implemented without the output interleaving controller 1206 and the auxiliary-accept-stage arbiters 1208. Alternatively, if the input scheduler 204 is coupled to a decoupling circuit 208 that has the virtual-channel-queue structure 404 (as shown in FIG. 11-1), the input scheduler 204 can include the output interleaving controller 1206 and the auxiliary- accept-stage arbiters 1208. This enables the input scheduler 204 to perform a first arbitration that is closer to the initiators 302 using the grant-stage arbiters 1204 and a second arbitration that is closer to the targets 304 using the auxiliary-accept-stage arbiters 1208. Implementations of the grant-stage arbiters 1204 and the auxiliary-accept-stage arbiters 1208 can vary' depending on whether the input scheduling 310 is performed using the input-first mode 322 or the output-first mode 324, as further described below.

[0087] Consider an example implementation in which the input scheduler 204 of FIG. 11-1 operates using the input-first mode 322. In the input-first mode 322, arbitration at the grantstage 314 occurs per initiator 302 and is followed by arbitration at the auxiliary accept stage 316, which occurs per target 304. The grant-stage arbiters 1204 provide the arbitration for corresponding initiators 302 at the grant stage 314. The grant-stage arbiters 1204 can be implemented using multi-layered arbiters 1210. The multi-layered arbiters 1210 perform multilayered arbitration based on the traffic class, the virtual channels 806, the quality-of-service, and / or the ports.

[0088] In this example implementation, the auxiliary-accept-stage arbiters 1208 provide the arbitration for corresponding targets 304 at the auxiliary accept stage 316. The quantity of auxiliary-accept-stage arbiters 1208 is at least equal to the quantity of targets 304 (e.g., M). The auxiliary-accept-stage arbiters 1208 can be implemented using per- virtual -channel port arbiters 1212 (per-VC port arbiters 1212). The per-virtual-channel port arbiters 1212 perform arbitration based on the virtual channels 806. In other words, the per-virtual-channel port arbiters 1212 select one input port 306 for each virtual channel 806. Alternatively, the input scheduler 204 can be designed to perform input scheduling 310 using the output-first mode 324, as further described with respect to FIG. 12-2.

[0089] FIG. 12-2 illustrates a second example implementation of the input scheduler 204. In this example, the input scheduler 204 is coupled to a decoupling circuit 208 that has the virtual- channel-queue structure 404 (as shown in FIG. 11-1). As such, the input scheduler 204 can include the output interleaving controller 1206 and the auxiliary-accept-stage arbiters 1208. In contrast to the input scheduler 204 of FIG. 12-1, the input scheduler 204 of FIG. 12-2 operates using the output-first mode 324.

[0090] In the output-first mode 324, arbitration at the grant stage 314 occurs per target 304 and is followed by arbitration at the auxiliary accept stage 316, which occurs per initiator 302. In this example, the grant-stage arbiters 1204 can be implemented using per-virtual-channel port arbiters 1212, and the auxiliary-accept-stage arbiters 1208 can be implemented using multilayered arbiters 1210.

[0091] In the input-first mode 322, shown in FIG. 12-1, and the output-first mode 324, shown in FIG. 12-2, the input scheduler 204 generates the grant 328 based on the requests 326 and the internal credit availability 508. Operations of the input interleaving controller 1202, the output interleaving controller 1206, the grant-stage arbiters 1204, and the auxiliary -accept-stage arbiters 1208 can be based on a previous grant 328, as shown in the feedback loop in FIGs. 12-1 and 12-2. The input scheduler 204 sends a read enable signal 1214 to the corresponding initiator 302 based on the grant 328. The read enable signal 1214 causes the initiator 302 to send the data330 to the decoupling circuit 208. An example implementation of the output scheduler 206 is further described with respect to FIG. 13.

[0092] FIG. 13 illustrates an example implementation of the output scheduler 206. In the depicted configuration, the output scheduler 206 includes an optional quality-of-service extractor 1302 (e.g., QoS extractor 1302), an output interleaving controller 1304, and accept-stage arbiters 1306. The quality-of-service extractor 1302 decodes quality-of-service information from the internal requests 510. The output interleaving controller 1304 provides output locking and request masking based on the credit availability 332 of the target 304. The accept-stage arbiters 1306 provide arbitration at the accept stage 318.

[0093] If the output scheduler 206 is coupled to a decoupling circuit 208 having the virtual-input- queue structure 404 (as shown in FIG. 10), the accept-stage arbiters 1306 can be implemented using multi-layered arbiters 1308. The multi-layered arbiters 1308 are similar to the multi-layered arbiters 1210 and perform multi-layered arbitration based on the traffic class, the virtual channels 806, the quality-of-service, and / or the ports.

[0094] If the output scheduler 206 is coupled to a coupling circuit 208 having the virtual-channel - queue structure 404 (as shown in FIG. 11-1), the accept-stage arbiters 1306 can be implemented using stream arbiters 1310. The stream arbiters 1310 can perform arbitration based on the stream, which can include a combination of the traffic classes and the virtual channels. The stream arbiter 1310 can be less complex to implement compared to the multi-layered arbiter 1308.

[0095] FIG. 14 illustrates an example implementation of the decoupling circuit 208 to implement aspects of decoupling the grant and accept stages for scheduling. In the depicted configuration, the decoupling circuit 208 is shown to include a per-target set of buffers 802 corresponding to a target 304. The per-target set of buffers 802 includes the skid buffers 402, which can be organized based on the virtual -input-queue structure 404 or the virtual-channel-queue structure 406, as shown in FIGs. 8 and 9.

[0096] The decoupling circuit 208 can also include upstream selection circuits 1402-1 to 1402 -V if the decoupling circuit 208 has the virtual-channel-queue structure 406. In an example implementation, the upstream selection circuits 1402-1 to 1402-V are implemented using multiplexers. The decoupling circuit 208 can be implemented without the upstream selection circuits 1402-1 to 1402-V if the decoupling circuit 208 has the virtual-input-queue structure 404.

[0097] The decoupling circuit 208 also includes a downstream selection circuit 1404, which can also be implemented using a multiplexer. The decoupling circuit 208 can optionally include at least one storage circuit 1406. Although not explicitly shown due to space constraints, the decoupling circuit 208 can also include other per-target set of buffers 802 corresponding to othertargets 304. These other per-target set of buffers 802 can be similarly coupled to other upstream selection circuits 1402, other downstream selection circuits 1404, and / or other storage circuits 1406.

[0098] The upstream selection circuits 1402-1 to 1402-V enable data 330 provided by the initiators 302 to be written to the per-target set of buffers 802 as part of the input scheduling 310. The upstream selection circuits 1402-1 to 1402-V identify which initiators 302 and which beats 702 are to be passed to the per-target set of buffers 802 based on the grant 328. In this example, each upstream selection circuit 1402 is associated with a virtual channel 806. For example, the upstream selection circuits 1402-1 to 1402-V respectively correspond to the virtual channels 806-1 to 806-V. Each upstream selection circuit 1402 passes the data 330 that is associated with its corresponding virtual channel 806 from one of the initiators 302 to the per- target set of buffers 802 based on the grant 328.

[0099] The downstream selection circuit 1404 enables data 336 stored within the per-target set of buffers 802 to be read and passed to the target 304 as part of the output scheduling 312. In this example, the data 336 can be associated with different initiators 302 and different virtual channels 806. The downstream selection circuit 1404 passes the data 336 corresponding to the internal grant 512.

[0100] Some implementations of the decoupling circuit 208 can include a storage circuit 1406, which is coupled between the downstream selection circuit 1404 and the output port 308. The storage circuit 1406 can be implemented using at least one flip-flop. In some cases, the storage circuit 1406 is implemented using multiple flip-flops to form a pipeline. In this case, the storage circuit 1406 can be referred to as a pipe stage. The storage circuit 1406 provides timing isolation between the output scheduling 312 and further downstream logic implemented at the target 304. This isolation can address any delays or disruptions between the output scheduling 312 and the downstream logic. With the storage circuit 1406, the data 336 can propagate from the per-target set of buffers 802 to the storage circuit 1406 in a single cycle 620. On one or more subsequent cycles 620. the data 336 can propagate from the storage circuit 1406 to the target 304. To support the timing isolation, the skid buffers 402 can be implemented with a minimum depth of two. In general, the depth of the skid buffers 402 can depend on the quantify of timing stages within the storage circuit 1406.Example Methods

[0101] FIGs. 15 and 16 depict example methods 1500 and 1600 for implementing aspects of scheduling using decoupled grant and accept stages. Methods 1500 and 1600 are shown as a setof operations (or acts) performed but not necessarily limited to the order or combinations in which the operations are shown herein. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and / or alternate methods. In portions of the following discussion, reference may be made to the environment 100 of FIG. 1, and entities detailed in FIGs. 2 to 4, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.

[0102] At 1502 in FIG. 15, input scheduling is performed during a first cycle. The input scheduling transfers data from at least one initiator to a decoupling circuit of the arbiter. The multiple initiators are coupled to multiple input ports of the arbiter. For example, the arbiter 202 performs input scheduling 310 during a first cycle 620-1, as described with respect to FIG. 6. The input scheduling 310 enables the data 330 (e.g., a beat 702) to be transferred from at least one initiator 302 to the decoupling circuit 208, as shown in FIGs. 3 to 5. The arbiter 202 has multiple input ports 306, which are coupled to corresponding initiators 302, as shown in FIG. 4.

[0103] At 1504, output scheduling is performed during a second cycle. The output scheduling transfers data from the decoupling circuit to at least one target of multiple targets. The multiple targets are coupled to multiple output ports of the arbiter. For example, the arbiter 202 performs output scheduling 312 during a second cycle 620-2, as described with respect to FIG. 6. The output scheduling 312 enables the data 336 (e.g., the beat 702) to be transferred from the decoupling circuit 208 to at least one target 304, as shown in FIGs. 3 to 5. The arbiter 202 has multiple output ports 308, which are coupled to corresponding targets 304, as shown in FIG. 4.

[0104] Although the above method is described with respect to at least one initiator 302 and at least one target 304, it is to be understood that the arbiter 202 can be performing input scheduling 310 and output scheduling 312 with respect to multiple initiators 302 and multiple targets 304. For example, the arbiter 202 can perform, during the first cycle 620-1, input scheduling 310 to transfer data (e.g., different beats 702) from qualified initiators 302 to the decoupling circuit 208. Additionally, the arbiter 202 can perform, during the second cycle 620-2, output scheduling 312 to transfer the data (e.g.. the different beats 702) from the decoupling circuit 208 to different targets 304.

[0105] At 1602 in FIG. 16, output scheduling is performed during a first cycle to transfer a first beat from a decoupling circuit of an arbiter to a first target. For example, the arbiter 202 performs output scheduling 312 during a first cycle 620-1 (or during a first time interval 704-1) to transfer the first beat 702-1 from the decoupling circuit 208 to a target 304, as shown in FIG. 7.

[0106] At 1604, input scheduling is performed during the first cycle to transfer a second beat from a first initiator to the decoupling circuit. For example, the arbiter 202 performs inputscheduling 310 during the first cycle 620-1 (or during the first time interval 704-1) to transfer the second beat 702-2 from the initiator 302 to the decoupling circuit 208, as shown in FIG. 7. By performing the methods 1500 and 1600 the arbiter 202 can be implemented with a scalable architecture 114 and perform high-speed, high-throughput scheduling 116.

[0107] Although the above method is described with respect to a first target 304 and a first initiator 302, it is to be understood that the arbiter 202 can be performing output scheduling 312 and input scheduling 310 with respect to multiple targets 304 and multiple initiators 302. For example, the arbiter 202 can perform, during the first cycle 620-1, output scheduling 312 to transfer data 336 (e.g., multiple beats 702) from the decoupling circuit 208 to multiple targets 304. In various situations, the multiple targets 304 can include all of the targets 304-1 to 304-M or a subset of the targets 304-1 to 304-M. The arbiter 202 can also perform, during the first cycle 620-1, input scheduling 310 to transfer other data 330 (e.g., other beats 702) from multiple initiators 302 to the decoupling circuit 208. In various situations, the multiple initiators 302 can include all of the initiators 302-1 to 302-N or a subset of the initiators 302-1 to 302-N.Example Computing System

[0108] FIG. 17 illustrates various components of an example computing system 1700 that can be implemented as any type of client, server, and / or computing device as described with reference to the previous FIGs. 2 and 3 to implement aspects of scheduling using decoupled grant and accept stages.

[0109] The computing system 1700 includes communication devices 1702 that enable wired and / or wireless communication of device data 1704 (e.g., received data, data that is being received, data scheduled for broadcast, or data packets of the data). The device data 1704 or other device content can include configuration settings of the device, media content stored on the device, and / or information associated with a user of the device. Media content stored on the computing system 1700 can include any type of audio, video, and / or image data. The computing system 1700 includes one or more data inputs 1706 via which any type of data, media content, and / or inputs can be received.

[0110] The computing system 1700 also includes communication interfaces 1708, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and as any other type of communication interface. The communication interfaces 1708 provide a connection and / or communication links between the computing system 1700 and a communication network by which other electronic, computing, and communication devices communicate data with the computing system 1700.

[0111] The computing system 1700 includes one or more processors 1710 (e.g., any of microprocessors, controllers, and the like), which process various computer-executable instructions to control the operation of the computing system 1700. Alternatively or in addition, the computing system 1700 can be implemented with any one or combination of hardware, firmware, or fixed logic circuitry that is implemented in connection with processing and control circuits which are generally identified at 1712. Although not shown, the computing system 1700 can include a system bus or data transfer system that couples the various components within the device. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus. and / or a processor or local bus that utilizes any of a variety of bus architectures.

[0112] The computing system 1700 also includes a computer-readable medium 1714 (CRM 1714), such as one or more memory devices that enable persistent and / or non-transitory data storage (i.e., in contrast to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of a read-only memory' (ROM), flash memory, EPROM, EEPROM, etc ), and a disk storage device. The disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and / or rewriteable compact disc (CD), any type of a digital versatile disc (DVD), and the like. The computing system 1700 can also include a mass storage medium device (storage medium) 1716.

[0113] The computer-readable medium 1714 provides data storage mechanisms to store the device data 1704, as well as various device applications and any other types of information and / or data related to operational aspects of the computing system 1700. For example, an operating system can be maintained as a computer application with the computer-readable medium 1714 and executed on the processors 1710. The device applications may' include a device manager, such as any form of a control application, software application, signal-processing and control module, code that is native to a particular device, a hardware abstraction layer for a particular device, and so on.

[0114] The computing system 1700 also includes at least one system-on-chip 106. The systemon-chip 106 includes the subsystems 108-1 to 108-S and the arbiter 202. In some implementations, the processor 1710, the processing and control 1712, the computer-readable medium 1714, and / or the storage medium 1716 can be implemented using one or more of the subsystems 108. The arbiter 202 can perform aspects of scheduling using decoupled grant and accept stages. With this, the arbiter 202 can implement a scheduler, such as the crossbarscheduler 1 12, that has the scalable architecture 114 and can perform high-speed, high-throughput scheduling 116, as described with respect to FIG. 1.Conclusion

[0115] Although techniques using, and apparatuses including, scheduling using decoupled grant and accept stages have been described in language specific to features and / or methods, it is to be understood that the subj ect of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of scheduling using decoupled grant and accept stages.

[0116] Some Examples are described below.

[0117] Example 1: A method performed by an arbiter, the method comprising: performing, during a first cycle, input scheduling to transfer data from at least one initiator of multiple initiators to a decoupling circuit of the arbiter, the multiple initiators coupled to multiple input ports of the arbiter; and performing, during a second cycle, output scheduling to transfer the data from the decoupling circuit to at least one target of multiple targets, the multiple targets coupled to multiple output ports of the arbiter.

[0118] Example 2: The method of example 1 , wherein the transferring of the data from the at least one initiator to the decoupling circuit comprises transferring multiple beats stored by the multiple initiators to multiple skid buffers of the decoupling circuit.

[0119] Example 3: The method of example 2, wherein: the decoupling circuit has a virtual -input-queue structure; and each skid buffer of the multiple skid buffers is associated with one of the multiple targets, one of the multiple initiators, and a virtual channel.

[0120] Example 4: The method of example 2, w herein: the decoupling circuit has a virtual-channel-queue structure; and each skid buffer of the multiple skid buffers is associated with one of the multiple targets and a virtual channel.

[0121] Example 5 : The method of any previous example, wherein the transferring of the data from the decoupling circuit to the at least one target comprises transferring multiple beats stored by the decoupling circuit to the multiple targets, the multiple beats corresponding to the multiple targets.

[0122] Example 6: The method of any previous example, wherein the performing of the input scheduling comprises: receiving, by an input scheduler of the arbiter, multiple requests from a first initiator of the multiple initiators; granting, by the input scheduler, one request of the multiple requests, the one request representing a request to forward a beat to a first target of the multiple targets, the beat representing at least a portion of the data; and storing, by a skid buffer of the decoupling circuit, the beat.

[0123] Example 7: The method of example 6, wherein: the performing of the input scheduling further comprises receiving, by the input scheduler, internal credit availability provided by the decoupling circuit, the internal credit availability indicating an availability of the skid buffer; and the granting of the one request comprises granting the one request based on the internal credit availability provided by the decoupling circuit.

[0124] Example 8: The method of example 6 or 7, wherein the performing of the output scheduling comprises: receiving, by an output scheduler of the arbiter, multiple internal requests sent by the decoupling circuit; granting, by the output scheduler, one internal request of the multiple internal requests, the one internal request representing a request to forward the beat from the skid buffer to the first target; and sending, by the skid buffer and based on the granting of the internal request, the beat to a first output port of the multiple output ports, the first output port coupled to the first target.

[0125] Example 9: The method of example 8, wherein: the performing of the output scheduling comprises receiving, by the output scheduler, credit availability provided by the first target, the credit availability indicating an availability7of the first target to receive the data; and the granting of the internal request comprises granting the one internal request based on the credit availability7of the first target.

[0126] Example 10: The method of any previous example, further comprising: performing, during the second cycle, input scheduling to transfer other data from the at least one initiator to the decoupling circuit.

[0127] Example 11 : The method of example 10, wherein: the performing of the input scheduling during the second cycle comprises executing a grant stage of the arbiter; the performing of the output scheduling during the second cycle comprises executing an accept stage of the arbiter; and the executing of the grant stage and the accept stage comprises executing the grant stage and the accept stage in parallel.

[0128] Example 12: The method of example 11, wherein: the performing of the input scheduling during the second cycle comprises executing the grant stage and an auxiliary accept stage of the arbiter in series during the second cycle based on an input-first mode or an output-first mode.

[0129] Example 13: The method of example 12, wherein the executing of the grant stage and the auxiliary accept stage based on the input-first mode comprises: performing arbitration on a full basis to execute the grant stage; and performing arbitration on a per-virtual-channel basis to execute the auxiliary accept stage.

[0130] Example 14: The method of example 12, wherein the executing of the grant stage and the auxiliary accept stage based on the output-first mode comprises: performing arbitration on a per-virtual-channel basis to execute the grant stage; and performing arbitration on a full basis to execute the auxiliary accept stage.

[0131] Example 15: A method performed by an arbiter, the method comprising: performing, during a first cycle, output scheduling to transfer a first beat from a decoupling circuit of the arbiter to a first target; and performing, during the first cycle, input scheduling to transfer a second beat from a first initiator to the decoupling circuit.

[0132] Example 16: The method of example 15, further comprising: reading the first beat from a skid buffer of the decoupling circuit during the first cycle; and writing the second beat to the skid buffer during the first cycle.

[0133] Example 17 : The method of example 15 or 16, wherein the decoupling circuit has a virtual- input-queue structure or a virtual-channel-queue structure.

[0134] Example 18: The method of any one of examples 15 to 17, further comprising: performing, during a cycle that occurs prior to the first cycle, the input scheduling to transfer the first beat from the first initiator to the decoupling circuit.

[0135] Example 19: The method of any one of examples 15 to 18, wherein: the performing of the output scheduling during the first cycle comprises executing an accept stage of the arbiter; the performing of the input scheduling during the first cycle comprises executing a grant stage of the arbiter; and the executing of the accept stage and the grant stage comprises executing the accept stage and the grant stage in parallel.

[0136] Example 20: The method of example 19. wherein the performing of the output scheduling during the first cycle comprises executing the grant stage and an auxiliary accept stage of the arbiter in series during the first cycle.

[0137] Example 21 : An apparatus comprising: a crossbar scheduler comprising an arbiter configured to perform any one of the methods of examples 1 to 12 or any one of the methods of examples 15 to 20.

[0138] Example 22: The apparatus of example 21, wherein the arbiter comprises: an input scheduler; and an output scheduler; and a decoupling circuit coupled between the input scheduling and the output scheduler.

[0139] Example 23: The apparatus of example 22, wherein: the decoupling circuit comprises multiple skid buffers; and the decoupling circuit has a virtual-input-queue structure or a virtual-channel-queue structure.

[0140] Example 24: A computer-readable storage medium comprising instructions that, responsive to execution by an arbiter, cause the arbiter to perform any one of the methods of examples 1 to 12 or any one of the methods of examples 15 to 20.

Claims

CLAIMSWhat is claimed is:

1. A method performed by an arbiter, the method comprising: performing, during a first cycle, input scheduling to transfer data from at least one initiator of multiple initiators to a decoupling circuit of the arbiter, the multiple initiators coupled to multiple input ports of the arbiter; and performing, during a second cycle, output scheduling to transfer the data from the decoupling circuit to at least one target of multiple targets, the multiple targets coupled to multiple output ports of the arbiter.

2. The method of claim 1 , wherein the transferring of the data from the at least one initiator to the decoupling circuit comprises transferring multiple beats stored by the multiple initiators to multiple skid buffers of the decoupling circuit.

3. The method of claim 2, wherein: the decoupling circuit has a virtual-input-queue structure; and each skid buffer of the multiple skid buffers is associated with one of the multiple targets, one of the multiple initiators, and a virtual channel.

4. The method of claim 2, wherein: the decoupling circuit has a virtual-channel-queue structure; and each skid buffer of the multiple skid buffers is associated with one of the multiple targets and a virtual channel.

5. The method of any previous claim, wherein the transferring of the data from the decoupling circuit to the at least one target comprises transferring multiple beats stored by the decoupling circuit to the multiple targets, the multiple beats corresponding to the multiple targets.

6. The method of any previous claim, wherein the performing of the input scheduling comprises: receiving, by an input scheduler of the arbiter, multiple requests from a first initiator of the multiple initiators; granting, by the input scheduler, one request of the multiple requests, the one request representing a request to forward a beat to a first target of the multiple targets, the beat representing at least a portion of the data; and storing, by a skid buffer of the decoupling circuit, the beat.

7. The method of claim 6, wherein: the performing of the input scheduling further comprises receiving, by the input scheduler, internal credit availability provided by the decoupling circuit, the internal credit availability indicating an availability of the skid buffer; and the granting of the one request comprises granting the one request based on the internal credit availability provided by the decoupling circuit.

8. The method of claim 6 or 7, wherein the performing of the output scheduling comprises: receiving, by an output scheduler of the arbiter, multiple internal requests sent by the decoupling circuit; granting, by the output scheduler, one internal request of the multiple internal requests, the one internal request representing a request to forward the beat from the skid buffer to the first target; and sending, by the skid buffer and based on the granting of the internal request, the beat to a first output port of the multiple output ports, the first output port coupled to the first target.

9. The method of claim 8, wherein: the performing of the output scheduling comprises receiving, by the output scheduler, credit availability provided by the first target, the credit availability indicating an availability of the first target to receive the data; and the granting of the internal request comprises granting the one internal request based on the credit availability of the first target.

10. The method of any previous claim, further comprising: performing, during the second cycle, input scheduling to transfer other data from the at least one initiator to the decoupling circuit.

11. The method of claim 10, wherein: the performing of the input scheduling during the second cycle comprises executing a grant stage of the arbiter; the performing of the output scheduling during the second cycle comprises executing an accept stage of the arbiter; and the executing of the grant stage and the accept stage comprises executing the grant stage and the accept stage in parallel.

12. The method of claim 11, wherein: the performing of the input scheduling during the second cycle comprises executing the grant stage and an auxiliary accept stage of the arbiter in series during the second cycle based on an input-first mode or an output-first mode.

13. A method performed by an arbiter, the method comprising: performing, during a first cycle, output scheduling to transfer a first beat from a decoupling circuit of the arbiter to a first target; and performing, during the first cycle, input scheduling to transfer a second beat from a first initiator to the decoupling circuit.

14. The method of claim 13, further comprising: reading the first beat from a skid buffer of the decoupling circuit during the first cycle; and writing the second beat to the skid buffer during the first cycle.

15. The method of claim 13 or 14, wherein the decoupling circuit has a virtual -input-queue structure or a virtual-channel-queue structure.

16. The method of any one of claims 13 to 15. further comprising: performing, during a cycle that occurs prior to the first cycle, the input scheduling to transfer the first beat from the first initiator to the decoupling circuit.

17. The method of any one of claims 13 to 16, wherein: the performing of the output scheduling during the first cycle comprises executing an accept stage of the arbiter; the performing of the input scheduling during the first cycle comprises executing a grant stage of the arbiter; and the executing of the accept stage and the grant stage comprises executing the accept stage and the grant stage in parallel.

18. The method of claim 17, wherein the performing of the output scheduling during the first cycle comprises executing the grant stage and an auxiliary accept stage of the arbiter in series during the first cycle.

19. An apparatus comprising: a crossbar scheduler comprising an arbiter configured to perform any one of the methods of claims 1 to 12 or any one of the methods of claims 13 to 18.

20. A computer-readable storage medium comprising instructions that, responsive to execution by an arbiter, cause the arbiter to perform any one of the methods of claims 1 to 12 or any one of the methods of claims 13 to 18.

Images