Including dedicated accelerators in graph nodes

Abstract

Systems, apparatuses, and methods for implementing hierarchical scheduling in a fixed function graphics pipeline are disclosed. In various implementations, a processor includes a pipeline including a plurality of fixed function units, and a scheduler. The scheduler is configured to: in response to a first mode of operation, schedule a first operation by scheduling the first operation with a first unit of the pipeline for execution by one or more fixed function units of the pipeline; and in response to the second mode of operation, scheduling the second operation for execution by the selected fixed function unit of the pipeline by scheduling the second operation directly to the selected fixed function unit independent of the sequential arrangement of one or more fixed function units in the pipeline.

Description

Background Art

[0001] Description of Related Technologies

[0002] The graphics processing pipeline performs a series of steps to convert input geometry into an image for display on a screen. In some examples, these steps are specified as commands to be executed by a host (such as a central processing unit running an application). Many rendering workloads require large command sequences, and thus generating and processing these sequences is a complex task.

[0003] Generally, the graphics processing pipeline consists of fixed-function units that are configured to perform a set of specialized functions in a graphics processing unit designed to support them. Each fixed-function unit typically has a programming abstraction that has a series of well-defined and specifically named graphics pipeline stages. However, in a graphics pipeline that includes multiple fixed-function units integrated into the pipeline, each unit performs a set task in a sequential manner, and there is no way to know the state of a particular intermediate task until the operation has traversed the entire pipeline.

[0004] In view of the above, there is a need for improved systems and methods for controlling a graphics processing pipeline having fixed-function units. Brief Description of the Drawings

[0005] The advantages of the methods and mechanisms described herein can be better understood by reference to the following description in conjunction with the accompanying drawings, in which:[

[0006] Figure 1 is a block diagram of a particular implementation of a computing system.

[0007] Figure 2 is a block diagram of a parallel processor that illustrates a hierarchical scheduler.

[0008] Figure 3 is a block diagram that shows the details of an exemplary graphics processing pipeline.

[0009] Figure 4A and Figure 4B illustrate an example scheduler graph.

[0010] Figure 5 is a block diagram of a control logic component that illustrates a hierarchical scheduling for implementing a pipeline.

[0011] Figure 6A and Figure 6B illustrate an exemplary graphics processing pipeline executed in different operating modes.

[0012] Figure 7 is a block diagram of an exemplary data flow configuration of a graphics processing pipeline for implementing a shader.

[0013] Figure 8 Illustrates a method for scheduling operations into one or more fixed function units in a pipeline. Detailed Description

[0014] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one of ordinary skill in the art should recognize that various specific implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the methods described herein. It should be understood that, for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some of these elements may be exaggerated relative to other elements.

[0015] Systems, apparatuses, and methods for implementing scheduling of fixed function units in a pipeline are disclosed. In various specific implementations, a processor includes a pipeline that includes one or more fixed function units, control logic components, and a scheduler. The scheduler is configured to schedule one or more operations into the pipeline using a pipeline mode (first mode) or a direct access mode (second mode). In the pipeline mode, an operation scheduled for execution by a fixed function unit of the pipeline enters the pipeline at the head, regardless of the position of the fixed function unit within the pipeline, and has a result that exits at the end of the pipeline. In the direct access mode, the control logic circuitry can use data from the scheduler to schedule the operation for execution by a selected fixed function unit of the pipeline by directly scheduling the operation to the selected fixed function unit without having to enter at the head of the pipeline or otherwise traverse other fixed function units of the pipeline.

[0016] Now referring to Figure 1 , a block diagram of a specific implementation of a computing system 100 is shown. In one specific implementation, the computing system 100 includes at least processors 105A-N, a control unit 110, an input / output (I / O) interface 120, a bus 125, a memory controller 130, a network interface 135, a memory device 140, a power supply 145, a power management unit 150, a display controller 160, and a display 165. In other specific implementations, the computing system 100 includes other components and / or the computing system 100 is arranged differently. Processors 105A-N represent any number of processors included in the system 100, where the number of processors varies according to the specific implementation.

[0017] In a specific implementation, the processor 105A is a general-purpose processor, such as a central processing unit (CPU). In a specific implementation, the processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. In a specific implementation, the processor 105N is a GPU that provides pixels to the display controller 160 for driving to the display 165. In some specific implementations, the processors 105A-N include multiple data parallel processors. In a specific implementation, the control unit 110 is a software driver executed on the processor 105A. In other specific implementations, the control unit 110 includes control logic components independent of and / or incorporated within the processors 105A-N. Generally speaking, the control unit 110 is any suitable combination of software and / or hardware.

[0018] The memory controller 130 represents any number and type of memory controllers accessible by the processors 105A-N. The memory controller 130 is coupled to any number and type of memory devices 140. The memory devices 140 represent any number and type of memory devices. For example, the types of memory in the memory devices 140 include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc.

[0019] The I / O interface 120 represents any number and type of I / O interfaces (e.g., Peripheral Component Interconnect (PCI) bus, PCI-Extended (PCI-X), PCI Express (PCIe) (PCI High Speed) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices (not shown) are coupled to the I / O interface 120. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, media recording devices, external storage devices, network interface cards, etc. The network interface 135 is used to receive and send network messages over a network. The bus 125 represents any type of bus or fabric having any number of links for connecting different components of the system 100 together.

[0020] In a specific implementation, queue 142 is located in memory device 140. In other specific implementations, queue 142 is stored in other locations within system 100. Queue 142 represents any number and type of queues allocated within system 100. In a specific implementation, queue 142 stores rendering tasks to be performed for frames being rendered. In a specific implementation, based on input received via network interface 135, the rendering tasks are queued in queue 142. For example, in one scenario, the input is generated by a user of a video game application and sent to system 100 via a network (not shown). In another specific implementation, the input is generated by a peripheral device connected to I / O interface 120.

[0021] In a specific implementation, power management unit 150 manages the power supply from power source 145 to the components of system 100, and power management unit 150 controls the various power performance states of the components within system 100. In response to receiving an update from control unit 110, power management unit 150 causes other components within system 100 to increase or decrease their current power performance states. In various specific implementations, changing the power performance state includes changing the current operating frequency of the device and / or changing the current voltage level of the device. When the power performance state of processors 105A-N is decreased, this typically results in taking longer to complete the computational tasks that processors 105A-N are performing.

[0022] In a specific implementation, control unit 110 sends a command to power management unit 150 to cause one or more of processors 105 to operate in a relatively high power performance state in response to determining that the number of tasks of the processor exceeds a threshold, a specific quality of service requirement needs to be met, etc.

[0023] In various specific implementations, computing system 100 is any one of a computer, a desktop computer, a mobile device, a server, or various other types of computing systems or devices. It should be noted that the number of components of computing system 100 varies depending on the specific implementation. For example, in other specific implementations, there are more or fewer of each component than Figure 1 shown. It should also be noted that in other specific implementations, computing system 100 includes Figure 1 other components not shown in Figure 1 and / or omits one or more of the components shown in computing system 100. Additionally, in other specific implementations, computing system 100 is constructed in a manner different from

[0024] Now turning to Figure 2, shows a parallel processor 200 that implements a hierarchical scheduling domain. In a particular implementation, the parallel processor 200 includes a plurality of scheduling domains 204. Each scheduling domain 204 corresponds to a shader engine. As shown, each shader engine 204 includes a plurality of workgroup processors (WGP) 208, where each workgroup processor includes one or more computing units (not shown). Each of the shader engines 208 is configured to execute a plurality of work items received from a command processor (also referred to as a "global scheduler") 214 external to the scheduling domain 204. In a particular implementation, each scheduling domain also includes a local work graph scheduler (WGS) 206 (or "scheduler") and a local cache 210. In various particular implementations, although each of the shader engines 204 includes a local cache 210, the cache 212 is shared by the shader engines 204. In this way, data can be communicated between the shader engines 204. In an exemplary particular implementation, each of the WGS and the global scheduler can access a separate mailbox that a given entity can use to communicate with another entity in the system without using the main memory subsystem of the parallel processor 200. In one example, the dedicated mailbox of the global scheduler can be located in the cache 212. Additionally, each WGS 206 can also have a dedicated mailbox, which in one particular implementation can be located in the cache 210 associated with the WGS 206. Other possible locations for the dedicated mailbox are conceivable and within the scope of the present disclosure.

[0025] In a particular implementation, the WGS 206 is configured to directly access the local cache 210, thereby eliminating the need to communicate through a higher-level scheduling hierarchy. In this way, scheduling latency is reduced, and finer-grained scheduling can be achieved. That is, the WGS 206 can schedule work items to one or more WGP 208 more quickly and more on a local basis. Additionally, the structure of the shader engines 204 enables a single WGS 206 to be used for each shader 204, making it easier to scale the shader engines 204. For example, since each of the shader engines 204 is configured to perform local scheduling, additional shader engines can be easily added to the processor.

[0026] In one particular implementation, a given shader 204 can be used to control one or more fixed-function units by exposing a given fixed-function unit to the graphics scheduling mechanism via the scheduler 206. The one or more fixed-function units can be, for example, a graphics processing pipeline that includes a plurality of fixed-function units (e.g., Figure 3Part of the graphics pipeline 300), or part of other processing pipelines composed of one or more fixed-function units. In the following discussion, for the purpose of discussion, the graphics pipeline can generally be referred to. However, the methods and mechanisms described herein are applicable to pipelines other than the graphics pipeline itself. For example, a given fixed-function unit can be externally controlled by the scheduler 206, such that the scheduler 206 can notify the fixed-function unit of one or more operations to be performed, and command the fixed-function unit to write data to a specific memory location (e.g., a dedicated mailbox) or system memory, the data being related to the execution results of one or more operations. Additionally, in another specific implementation, the scheduler 206 can also obtain information associated with the status of the fixed-function unit, the overloading or underloading of the fixed-function unit, the signal error condition associated with the fixed-function unit, etc.

[0027] In operation, to utilize the scheduler 206 to control the fixed-function unit, a given shader 204 can execute a pipeline of one or more fixed-function units (e.g., a graphics pipeline or another pipeline) in different operation modes. In a specific implementation, in the first operation mode, the scheduler 206 can schedule an operation for execution by the pipeline by scheduling an operation for a first fixed-function unit of the pipeline such that the output of the execution of the operation can be used as input by a second fixed-function unit of the pipeline, the second fixed-function unit being connected to the first fixed-function unit in a sequential manner. That is, in the first operation mode, before the scheduler 206 can determine the output of the execution of an operation, each operation performed needs to traverse all the fixed-function units of the pipeline.

[0028] However, in the second operation mode, the scheduler 206 can schedule an operation for a selected fixed-function unit of the pipeline independent of the sequence of fixed-function units in the pipeline. That is, in the second operation mode, the scheduler can directly assign an operation to be executed to the selected fixed-function unit without having to sequentially traverse all the units of the pipeline. To this end, the scheduler 206 utilizes the input data buffer of the selected fixed-function unit to notify the selected fixed-function unit of the execution of the operation. Additionally, once the operation is executed by the selected fixed-function unit, it can notify the scheduler 206 by utilizing the memory subsystem or by writing to a dedicated mailbox accessible by the scheduler 206. In a specific implementation, the scheduler 206 also notifies the selected fixed-function unit of where to send the data related to the execution results of the operation, e.g., to the input data buffer of another fixed-function unit or to a memory location accessible by another scheduler.

[0029] In a particular implementation, the fixed function units may also send push messages to the scheduler 206, and the messages may also be pulled by the scheduler 206 from the fixed function units. Whether a message is pushed or pulled may depend on the type of the message. For example, a message indicating the completion of the execution of operation 206 may be a push message sent from the fixed function unit to the scheduler 206. In addition, the scheduler 206 may pull messages indicating overload or underutilization of the fixed function units from the fixed function units.

[0030] In a particular implementation, the data streams to and from the fixed function units may be established by the scheduler 206 using a messaging protocol that may include messaging formats for different messages, such as messages for scheduling operations for execution, notifications that an operation has been executed by the fixed function unit, messages including status reports, etc.

[0031] Advantageously, implementing a pipeline allows for better control of the pipeline by controlling one or more fixed function units using nodes in the scheduling graph, while reducing the latency of executing one or more operations, since it may not be necessary to traverse all units of the pipeline for a given operation. In addition, exposing one or more fixed function units to memory locations associated with the shader may also facilitate an accurate understanding of the overload or underutilization of the fixed function units without having to wait for the execution of the entire pipeline, thereby improving the efficiency of operation scheduling.

[0032] Figure 3 is a block diagram illustrating an exemplary graphics processing pipeline. The graphics processing pipeline 300 includes one or more fixed function units each performing a specific function of the graphics processing pipeline 300. Each fixed function unit is partially or fully implemented as fixed function, non-programmable hardware external to the computing unit.

[0033] The input assembler 302 reads primitive data from a user-fill buffer (e.g., a buffer filled at the request of software (such as an application) executed by the processor 200), and assembles the data into primitives for use by the rest of the pipeline. Based on the primitive data included in the user-fill buffer, the input assembler 302 may generate different types of primitives. The input assembler 302 formats the assembled primitives for use by the rest of the pipeline.

[0034] The vertex shader 304 processes the vertices of the primitives assembled by the input assembler 302. The vertex shader 304 performs various per-vertex operations, such as transformation, skinning, morphing, and per-vertex lighting. Transformation operations include various operations for transforming vertex coordinates. These operations include modeling transformations, viewport transformations, projection transformations, perspective division, and viewport transformations that modify vertex coordinates, as well as one or more of other operations that modify non-coordinate attributes.

[0035] The vertex shader 304 is partially or fully implemented as a vertex shader program to be executed on one or more compute units. The vertex shader program is provided by the processor 200 and is based on a program pre-written by a computer programmer. A driver (not shown) compiles such a computer program to generate a vertex shader program having a format suitable for execution within the compute unit.

[0036] The hull shader 306, tessellator 308, and domain shader 310 work together to implement tessellation, which converts simple primitives into more complex primitives by subdividing the primitives. Based on the input primitives, the hull shader 306 generates patches for tessellation. The tessellator 308 generates a set of samples for the patches. The domain shader 310 calculates the vertex positions of the vertices corresponding to the samples of the patches. The hull shader 306 and domain shader 310 can be implemented as shader programs to be executed on the compute units, and these shader programs are compiled by the driver as in the case of the vertex shader 304.

[0037] The geometry shader 312 performs vertex operations on a per-primitive basis. A variety of different types of operations can be performed by the geometry shader 312, including operations such as: point sprite expansion, dynamic particle system operations, fur-fin generation, shadow volume generation, single-pass rendering to a cube map, per-primitive material swapping, and per-primitive material setting. In some cases, a geometry shader program compiled by the driver and executed on the compute unit performs the operations of the geometry shader 312.

[0038] The rasterizer 314 accepts simple primitives (triangles) generated upstream from the rasterizer 314 and rasterizes them. Rasterization includes determining which screen pixels (or sub-pixel samples) are covered by a particular primitive. Based on the upstream-generated primitives and the results of rasterization, the pixel shader 316 calculates the output values of the screen pixels. The pixel shader 316 can apply textures from the texture memory. The operations of the pixel shader 316 are performed by a pixel shader program that is compiled by the driver and executed on the compute unit.

[0039] The output merger 318 accepts the outputs from the pixel shader 316 and merges these outputs into the frame buffer, thereby performing operations such as z-testing and alpha blending to determine the final color of the screen pixels.

[0040] Typically, the scheduler 206 processes a workload by creating multiple "nodes" for scheduling using a scheduling graph. A "node" is a processing unit that accepts an input (e.g., from another node or as an input to the overall processing workload), processes the input, and generates an output. Some outputs include outputs to other nodes, and other outputs include the final output of the processing workload, such as an image to be displayed on a screen. Thus, the specific order of the nodes, the inputs used by those nodes, and the outputs generated by those nodes specify the overall processing workload. In a particular implementation, to perform operations by the fixed-function units of the pipeline 300, the scheduler 206 may generate one or more nodes with operations as inputs and schedule each of these nodes to a given fixed-function unit. The nodes may execute in a sequential manner in a first operating mode or in a selected non-sequential order in a second operating mode (as Figures 4A to 4B described). That is, based on the operating mode, the output of a node resulting from the execution of an operation by a first fixed-function unit may be used as an input to a second fixed-function unit of the pipeline in the first mode or as an input to a selected fixed-function unit in the second mode.

[0041] Now turning to Figure 4A , an example task graph 400 for scheduling operations for the graphics pipeline 300 is shown. The task graph 400 specifies the processing workload as a directed acyclic graph of a data buffer 402 (or "buffer 402") and nodes 404. As described above, the nodes 404 include specific work units, such as processing certain data by the graphics pipeline 300 configured in a fixed manner. The data buffer 402 includes data that serves as an input and / or an output of the nodes 404. The data buffer 402 can be an input to one node 404 and an output to another node, or can be an input or output to the overall processing workload (which corresponds to a single task graph). In a particular implementation, the buffer may include an image generated by a specific node 404 through the graphics pipeline 300 or data other than an image generated by the nodes 404 through the graphics pipeline 300.

[0042] The task graph 400 includes an arrow between a buffer 402 and a node 404. An arrow from the buffer 402 to the node 404 means that the node 404 accepts the data in the buffer 402 as input. An arrow from the node 404 to the buffer 402 means that the node 404 produces the data in the buffer 402 as output. Thus, the task graph 400 specifies the dependencies between the nodes 404. More specifically, a node 404 that accepts a buffer 402 generated by another node 404 as input must wait to execute until the buffer 402 has actually been generated. Thus, a node 404 that accepts a buffer 402 as input depends on other nodes 404 that produce those buffers 402 as output. If a third node 404 depends on a second node 404 and a first node 404 depends on the third node 404, then the first node 404 is also considered to depend on the second node 404. In other words, dependencies propagate through the arrows of the task graph 400. If neither depends on the other, then the two nodes 404 are independent.

[0043] As described above, the task graph specifying program is a program that specifies how to construct the task graph 400. Thus, the task graph specifying program indicates which nodes 404 will be performed and which buffers 402 are the inputs and outputs of the nodes 404. When making decisions about whether to include certain nodes 404 and / or buffers 402 in the task graph 400 and how the nodes 404 consume and / or produce the buffers 402, the task graph specifying program can consider runtime data, such as user-defined runtime data. Thus, the task graph specifying program is not just a static description of the task graph 400 evaluated by the processor 200. Instead, the task graph specifying program can variably construct the task graph 400 based on runtime parameters.

[0044] Now turning to Figure 4B , an example task graph 410 for scheduling operations for a graphics pipeline 300 using hierarchical scheduling is shown. In a particular implementation, as Figure 2One or more shaders 418 in the hierarchical scheduling structure described herein may use data buffers 412 and nodes 414 to directly schedule operations to fixed function units 416 independent of the sequential order of the units in pipeline 300. For example, shader 412A may schedule one or more operations for fixed function unit 416A by writing data to data buffer 412A as an input to node 414A, which is then fed to fixed function unit 416A for execution. In a particular implementation, scheduler 206 may write data to buffer 412A by mapping a particular memory or using direct memory access (DMA) to write data to buffer 412A. In another particular implementation, scheduler 206 may also send a pointer to data in a particular memory location, and fixed function unit 412A may read the data from that particular memory location.

[0045] As Figure 4B shown, in an exemplary particular implementation, the output generated by fixed function unit 416A may be accepted as an input into buffer 412B. In one particular implementation, the data in buffer 412B may be utilized based on the operating mode of processor 200. For example, in a first mode, the data in buffer 412B is used as an input to node 414B, which may be used to schedule operations for fixed function unit 416B, where fixed function unit 416B is the unit in the sequential order of pipeline 300 that is next to fixed function unit A. That is, in the first operating mode (i.e., pipeline mode), the output generated from a fixed function unit is used to generate a node that serves as an input to the next fixed function unit in the sequence of pipeline 300. When traversing the entire pipeline 300, the final output is read by a given shader engine of processor 200. In Figure 4B the example shown, pipeline 300 may include two fixed function units 416A and 416B, and once both of these fixed function units have completed execution of the operation (or the data otherwise traversed through the complete pipeline 300), the output is stored as data in buffer 412C, and shader 418B may access the output from this buffer. Although Figure 4B pipeline 300 is discussed as including two fixed function units, other configurations of pipeline 300 with more fixed function units are envisioned and within the scope of the present disclosure.

[0046] In a particular implementation, in a second operating mode (i.e., direct access mode), the output from a given fixed function unit may be accessed by a shader without having to traverse through all of the other fixed function units in pipeline 300. For example, as Figure 4BAs shown, the output from the fixed function unit 416A (as stored in buffer 412B) can be directly accessed by the shader 418A without traversing through the fixed function unit 412B and associated nodes and buffers. Additionally, in the second mode, the shader 418A can also notify the fixed function unit 416A of a given memory location (i.e., using the memory subsystem or a dedicated mailbox) to which the fixed function unit 416A can write data associated with the output. In such a scenario, the fixed function unit 416A is configured to send an indication to the shader 418A of the completion of execution of one or more operations scheduled by the shader 418A for the fixed function unit 416A. The fixed function unit 416A also reports its workload status, any error conditions, and other information to the shader 418A.

[0047] Using the second operating mode as described above, a given scheduler can control the data flow through the task graph 410B and ensure mitigation of issues associated with overloading a particular fixed function unit. Additionally, to communicate with each fixed function unit controlled by the scheduler, the scheduler utilizes a messaging protocol with each fixed function unit, which may include a message to start work, a notification that the work has been performed, and one or more statuses, as described above.

[0048] Now turning to Figure 5 , an exemplary graphics processing pipeline executed in different operating modes is shown. Figure 5 Illustrated in is pipeline 500, circuitry 503 associated with the direct access mode, and an enlarged view of a particular implementation of the direct access circuitry 503. In an implementation, a task graph (e.g., task graph 410) may identify a first operation that generates an output (and stored in a buffer, e.g., buffer 412) that can be consumed by a second operation. Based on the task graph, a scheduler (e.g., scheduler 206) may create one or more commands for scheduling the first operation into the pipeline 500, a command for storing the result of the first operation in a buffer (or memory location), and then create a command for executing the second operation that consumes data from the buffer and creates a new output. Scheduling of the pipeline 500 can be performed in "pipeline mode" 501 or "direct access mode" 503, as discussed below. Generally, in pipeline mode, data to be processed by one of the fixed function units (502A - 502N) enters the pipeline via input 504 at the head (or start) of the pipeline 500, and the result of the operation exits via output 506 at the tail (or end) of the pipeline 502. As used herein, a single subscripted numeral (e.g., 502) followed by a differentiating letter (e.g., A, B, C) may be collectively referred to by the subscripted numeral alone (e.g., fixed function unit 502).

[0049] In a particular implementation, the graphics pipeline 500 includes one or more fixed function units 502 (or "FFUs" 502) as shown. In a particular implementation, in pipeline mode, input operations can be scheduled for the pipeline 500 by queuing one or more operations in an input queue 504 or other location (e.g., scheduled by a scheduler using a scheduling graph as discussed above). Additionally, in pipeline mode, the output of an operation is generated at the end of the pipeline and stored in an output buffer 506. When operating in pipeline mode 501, one or more FFUs 502 can operate on the data. In various scenarios, not all FFUs operate on the received data. Instead, a signal or other indication can be provided to indicate whether a given FFU 502 is to operate on the received data. If a given FFU 502 does not operate on the received data, the FFU 502 can simply pass the received data to another FFU 502 (e.g., the next FFU 502 in the pipeline 500). However, when operating in pipeline mode, the data traverses the pipeline 500 before the output of a given operation is accessible.

[0050] For example, when operating in pipeline mode, the minimum latency for the execution of an operation includes at least the cumulative number of clock cycles for moving the received data through the pipeline (e.g., from the input buffer 504 and between each FFU 502), processing the data by the FFU 502 designated to process the data, and moving the processed data to the output queue 506 of the pipeline 500. Additionally, in various particular implementations, when operating in pipeline mode 501, the state of a given operation may be unknown to the scheduler (or other control circuitry), making it difficult to have visibility into when dependent operations may or may not be schedulable.

[0051] As an alternative to pipeline mode 501, in various particular implementations, a direct access mode 503 is available. In direct access mode 503, additional circuitry is implemented that allows direct access to one or more FFUs 502. How many FFUs 502 are provided with direct access is a matter of design choice. In Figure 5 FIG., an example of such additional circuitry 503 is shown. In the example shown, additional direct access circuitry 503 is shown that includes a control unit 508 and storage means for the input data to be processed ("In") and storage means for the output data of the processed data ("Out") (e.g., input and output buffers). With this additional circuitry, the scheduler 206 can directly schedule an operation to a selected FFU 502 without having to traverse the other FFUs 502 in the pipeline 500. To this end, as Figure 5As shown in the example in , the control unit 508 can receive data from the scheduler 206 for scheduling one or more operations, receive data directly from the cache or memory associated with the scheduler 206, store the data in the input queue (In), and select an operation from one or more operations for processing. When an operation is selected, the control unit 508 directly provides the received data to the FFU-3 502C without having to first traverse the FFUs 502A and 502B. That is, each time a new operation is to be scheduled for the FFU 502, the operation can be directly scheduled for the FFU 502 regardless of the execution of other FFUs 502 in the pipeline 500.

[0052] In addition, for each scheduled operation of the FFU 502C, the data associated with the output of the operation can also be directly accessed by the control unit 508 without waiting for the entire pipeline 500 to complete execution. In the example shown in the drawings, the control unit 508 can directly access the data associated with the output of the operation scheduled for the FFU 502C from a unit (e.g., from the output buffer Out) and store the data in its own output queue. Then, the scheduler 206 can access the data from the output queue of the control unit 508.

[0053] In one or more specific embodiments, the control unit 508 can further monitor one or more conditions associated with the operations of a given FFU 502. These conditions can include detecting the presence and / or quantity of data received in the input buffer (In), detecting the presence and / or quantity of data in the output buffer (Out), detecting the state of the corresponding FFU (e.g., the FFU is currently busy or not busy, the FFU is unavailable for a given period of time), the execution state of one or more operations (e.g., not started, in execution, completed, etc.), the overload or underutilization of the FFU 502, the error state of the FFU 502, control signals received from the scheduler 206, data associated with the operation mode, and others.

[0054] As Figure 5(Enlarged view of the direct access circuit 503 in the lower right corner) As shown, in some specific embodiments, the control unit 508 may also be connected to the multiplexer 510 such that the multiplexer 510 can feed inputs to the FFU 502 from a previous FFU in the pipeline 500 or directly from the input queue of the direct access circuit 503. In various specific embodiments, the control unit 508 provides signals to the multiplexer 510 to indicate which data to select based at least in part on the operating mode (pipeline mode or direct access mode). For example, the multiplexer may feed data from a previous FFU in the pipeline mode and feed data from the input queue of the control unit 508 in the direct access mode. In a specific embodiment, the control unit 508 may determine the operating mode based on metadata (or signals) received from the scheduler 206. For example, the control unit 508 may detect metadata associated with the data stored in the input buffer (In) or detect signals generated by the scheduler 206.

[0055] In some scenarios, the direct access mode 503 allows the scheduler 206 to directly access a single FFU. In another scenario, the direct access mode 503 allows the scheduler to operate the pipeline 500 in a hybrid mode, where both direct access and pipeline mode are used. For example, in such a scenario, the scheduler may schedule operations to be processed by two or more FFUs 502, including at least one direct access mode operation. In such a scenario, the scheduler may use the direct access mode 503 to directly access the first FFU 502 out of two or more FFUs. When performing a direct access, the scheduler may provide metadata (or signals) indicating which FFUs will be used in the operation. As an example, the scheduler 206 may provide a bit vector that includes bits for each FFU 502 in the pipeline. Then, such a bit vector may be provided as metadata along with the input data to a given FFU 502. By setting the corresponding bits, the scheduler 206 may indicate which FFUs 502 will process the data. In various scenarios, the indicated FFUs 502 may or may not be directly serially coupled in the pipeline 500. For example, if three FFUs 502 are to process data, the indicated FFUs 502 may be 502B, 502C, and 502D (i.e., directly serially). Alternatively, the indicated FFUs 502 may be 502B, 502D, 502F (i.e., some FFUs 502 in the pipeline 500 do not process). In these scenarios, the first access is a direct access to the initial FFU, while the remaining accesses are in pipeline mode performed by moving the data along the pipeline to the next FFU 502. It should also be noted that the direct access mode 503 can be used to retrieve the final data, rather than waiting for the processed data to traverse the pipeline to reach the output buffer 506. For example, in a given scenario, the scheduler may directly schedule an operation to FFU-2, where it is scheduled to be operated on by FFU-2 and FFU-3, but then exit the pipeline to the direct access mode buffer after FFU-3. These implementations and other implementations are possible and contemplated.

[0056] Now turning to Figure 6A, an exemplary operation of pipeline 600 in pipeline mode is shown. As described previously, the minimum latency of the operation execution in pipeline mode can be determined as the cumulative amount of clock cycles spent in the first FFU 502A to clock the data from input queue 504, the FFU 502A to process the data, and the output queue 506 at the end of pipeline 600 to clock the data associated with the output of the operation. Traditionally, in pipeline mode, due to the data generated by a given FFU 602 to execute an operation, it needs to traverse the entire pipeline 600 before it can be accessed as an input to one or more outstanding scheduled operations, regardless of which FFUs 602 are enabled or disabled within the pipeline 600 (e.g., bypassed or otherwise pass the data to another FFU without processing the data). In a specific implementation, the scheduler can utilize one or more control signals to schedule operations for the pipeline in pipeline mode, and the one or more control signals at least partially include data associated with the operation and metadata indicating which FFUs 602 need to process the data. For example, each FFU 602 that does not need to process the data can simply forward it to the next FFU 602 in pipeline 600. However, the data still needs to traverse the entire pipeline 600 before the scheduler can access it at the output queue.

[0057] In the example, it is assumed that the minimum latency of pipeline 600 is equal to N clock cycles, where N is a positive integer. Note that indexing of clock cycles is used herein for purposes of discussion, but other units of measurement are possible and contemplated. In the example shown in the figures, the minimum latency of the pipeline is 100 clock cycles. Note that clock cycles of other values (e.g., non-integer values) are contemplated and are within the scope of the present disclosure. In an exemplary implementation, the operations to be performed by any FFU 602 in pipeline 600 may consume a given number of data clock cycles, which may be fixed or may vary. In the example shown in Table 1, the fixed number of clock cycles for FFU 602A may be 50 clock cycles, the number of clock cycles for FFU 602B may be 40 clock cycles, and the number of clock cycles for FFU 602C to process data is 55 clock cycles. In the following example, it is assumed that there are three operations scheduled by the scheduler for execution by at least one FFU 602 of pipeline 600. When pipeline 600 operates in pipeline mode, a first operation scheduled for FFU 602A at N = 0 may produce an output that may be accessible at the end of the execution of the full pipeline 600, i.e., a minimum latency of at least N = 100. That is, before any output from a given FFU 602 is accessible, pipeline 600 is fully traversed. This includes the scheduling of the first operation at FFU 602A at N = 0, the processing of the first operation by FFU 602A (e.g., using 50 clock cycles), and finally the access to the output at the end of the execution of pipeline 600 at N = 100.

[0058] In one example, the output of the first operation performed by FFU 602A is used as the input to the outstanding scheduled operation corresponding to FFU 602B. In this case, when pipeline 600 is operating in pipeline mode, the earliest scheduling of the second operation of FFU 602B can occur at N = 100. As further depicted in Table 1, the second operation is scheduled at N = 100. For simplicity of discussion, the time that may be required to move the data output from pipeline 600 to a memory location before use in the second operation is ignored. In addition, since FFU 602B takes 40 clock cycles to process the data and the entire pipeline 600 needs to be traversed before accessing the output of the second operation, the output of the second operation is available only at N = 200. This is because the output of the first operation retrieved at N = 100 is fed back into the input queue of FFU 602A, taking time to move the data through or past FFU 602A, and then FFU 602B processes the data, which takes 40 cycles. Since this example assumes a minimum latency of 100 cycles, the earliest access to the output of this second operation is at N = 200 cycles. As shown in Table 1, the second operation is scheduled at N = 100 and the output is accessed only at N = 200. Continuing with this example, a later scheduled third operation will take at least 100 cycles. Thus, for this scenario, the shortest possible latency is 300 clock cycles.

[0059] Generally speaking, when pipeline 600 is operating in pipeline mode, it needs to be fully executed before any output can be accessed (i.e., data sequentially traverses through all FFUs 502). In some scenarios, a particular sequence of operations can be scheduled in the pipeline such that the data output from one FFU is used as the input to another FFU. However, when dependent operations are scheduled at different times as described above, the scheduler must wait for pipeline 600 to complete its execution, access the data, and use the data as input for scheduling another operation. In addition, when a given FFU 602 is not the first FFU 602 in pipeline 600, the scheduler must also wait for all previous FFUs 602 to process the data before an operation can be scheduled for that given FFU 602, resulting in an increased latency in the execution of the operation.

[0060] Now turning to Figure 6B, shows an exemplary operation of the pipeline 600 in direct access mode. Using the same example as above, the pipeline 600 has a minimum latency of 100 clock cycles for full traversal, the FFU 602A has a processing time of 50 clock cycles for processing data, the FFU 602B has a processing time of 40 clock cycles, and the FFU 602C has a processing time of 55 cycles. In direct access mode, at N = 0, the first operation is scheduled for the FFU 602A, which takes 50 clock cycles to process the data and produce an output. This is different from the pipeline mode, in which the output has an earliest availability at N = 100 cycles.

[0061] The control unit ( Figure 6B not shown in the figure) can store the data associated with the output of the first operation in its output buffer, which can be accessed by the scheduler. In the example shown in the figure, this output data can be used to schedule the second operation to the FFU 602B. In this case, the scheduler schedules the second operation for the FFU 602B at N = 50. As described above, to simplify the discussion, the time for moving data to and from the memory, etc. is ignored. As shown in Table 2, the first operation is scheduled at N = 0, the output of the first operation is accessed at N = 50, and at N = 50, the second operation is scheduled for the FFU 602B (using the data from the output of the first operation). In the same way, to schedule the third operation for the FFU 602C using the output from the second operation, the scheduler can directly access the data associated with the output of the second operation at N = 90, schedule the third operation at the same time point, and produce the final output at time N = 145, rather than Figure 6A producing the final output at N = 300 in

[0062] Therefore, such a system that directly accesses each FFU 602 can reduce the total latency of the pipeline 600. In addition, the control unit can continuously monitor the status of each FFU 602 during each execution cycle. In addition, with the direct access mode, the scheduler does not have to wait for an indication of the completion of the execution before starting the next execution.

[0063] Although Figure 6A and Figure 6BThe example in [description] depicts the FFU 502 being executed in a sequential manner, but in one or more specific embodiments, if there are no dependencies between the operations to be scheduled, the scheduler may schedule two or more FFUs 502 (i.e., for concurrent execution) in direct access mode. For example, at the start of execution, i.e., N = 0, the scheduler may schedule a first operation to FFU 602A and a second operation to FFU 602B such that the execution of these two operations is completed at N = 50. This may further reduce the total latency of the pipeline 600.

[0064] Now turning to Figure 7 , a block diagram is shown illustrating an exemplary data flow of a graphics processing pipeline for implementing a shader. In a specific embodiment, the pipeline may include fixed function units such as NIC 702, video decoder 706, machine learning unit (or "ML unit") 708, and video encoder unit 710. Each fixed function unit is controlled by a shader 704, such as shader 704A connected to NIC 702 and video decoder 706, shader 704B connected to ML unit 708 and video encoder 710, and shader 704C connected to video decoder 706 and video encoder 710.

[0065] In one specific embodiment, the NIC 702 may generate one or more work items at a fixed rate. For example, network packets including data to be processed are received over the network and may be queued by the NIC 702 for execution. The NIC 702 may notify the scheduler (not shown), e.g., as a push notification that a work item is ready for execution. In response, the scheduler accesses the incoming network packets and once a predetermined number of packets have been accumulated, the scheduler may notify shader 704A to start processing. In a specific embodiment, shader 704A generates an output based on the notification from the scheduler. For example, shader 704A may generate a second number of predetermined packets. Shader 704A may then notify the scheduler that the output has been generated.

[0066] The generated output from shader 704A can be input to video decoder 706. In a particular implementation, the scheduler can monitor the amount of work items input to each fixed function block, and if there is an overload in a given fixed function block, it does not initiate the next execution (different from the pipeline mode where no insight into the state of the fixed function unit may be available). Video decoder 706 can generate an output based on the input it receives, and the scheduler can receive a notification from video decoder 706 that the output has been generated and saved in a specific memory location. Based on this notification, the scheduler can read that specific memory location and store the generated output in a cache (not shown). Based on this output, video decoder 706 can forward a portion of the work items to ML unit 708 for execution, while the remaining work items can be forwarded by the scheduler to shader 704C.

[0067] In a particular implementation, based on a portion of the work items forwarded to ML unit 708, it can notify the scheduler, which in turn tells shader 704B when to start execution. Similarly, the scheduler schedules the remaining work items for video encoder 710 by notifying shader 704C. Video encoder 710 executes the received work items and creates a final output (e.g., an image or video to be presented at the user device), and notifies the scheduler of the completion of the execution. The scheduler can tell video encoder 710 the memory location to which the final output is to be sent, or in some cases discard the output. Additionally, once the execution is complete or the execution result is discarded, the scheduler can recycle each memory location accessed during the execution.

[0068] In a particular implementation, a fixed function unit (e.g., video decoder 706) can receive one or more work items from a given processor unit (not shown) instead of from a given scheduler, enabling the processor unit to drive the fixed function unit. For example, when the processor unit drives the fixed function unit, a given scheduler may only be able to throttle or configure these one or more work items. In the context of video decoder 706, for example, when the work items are related to the generation of video, the scheduler may be able to perform a resolution change to regulate the data rate. However, the initiation of the execution of the work items can still be controlled by an external processor unit.

[0069] Now turning to Figure 8, which shows an embodiment of a method 800 for controlling one or more fixed - function units in a graphics pipeline. As described previously, a control unit (e.g., control unit 508) may select an operation to be performed by the graphics pipeline (block 802). In a particular implementation, the graphics pipeline may include one or more fixed - function units, such as a geometry shader, an input assembler, a tessellator, a rasterizer, a video encoder, a video decoder, etc. For example, when the fixed - function unit is a geometry shader, the operation may be one of point sprite expansion, dynamic particle system operation, fur - fin generation, shadow volume generation, single - pass rendering to a cube map, per - primitive material swapping, per - primitive material setting, etc.

[0070] In a particular implementation, the control unit may select the operation for execution at least in part based on one or more control signals received from a scheduler. In an example, the control signals may include data associated with the operation and metadata indicating which fixed - function units need to process the data (i.e., whether to use pipeline mode or direct access mode). The control unit may store the data and associated metadata in its input queue and write the data and metadata to the input buffer of the fixed - function unit based on the operation mode. Once the operation is selected, the control unit may determine whether the fixed - function unit is to operate in a first operation mode (pipeline mode) or a second operation mode (direct access mode) (conditional block 804).

[0071] In the case of the first operation mode (the "first operation mode" branch of conditional block 804), the scheduler may notify the first fixed - function unit from the pipeline of the selected operation (block 806). In a particular implementation, in the first operation mode, the control unit determines, based on the data received from the scheduler, that the selected operation will be executed by the first fixed - function unit in a sequence of fixed - function units included in the pipeline. The data related to the execution of the selected operation is written to the input queue of the pipeline for the first fixed - function unit to execute (block 808).

[0072] The selected operation may then be executed by the first fixed - function unit, and the data associated with the output of the execution may traverse to each subsequent fixed - function unit of the pipeline. That is, in the first operation mode, the data related to the output of the selected operation is available for access only when the execution of the entire pipeline is complete. The control unit may read the data associated with the output from the output queue associated with the pipeline (block 810). In a particular implementation, this data from the output queue of the pipeline may be read by the control unit and stored in its own output queue, from which the scheduler may access the data.

[0073] However, in the case of the second operating mode (the "second operating mode" branch of conditional block 804), the control unit may notify a selected fixed-function unit in the sequence of units included in the pipeline (block 812). That is, in the second operating mode, the control unit may select a given fixed-function unit to perform a given operation (based on data received from the scheduler), regardless of the position of the fixed-function unit in the sequence of units in the pipeline. In a particular implementation, the selection of the fixed-function unit may be made based on the application using the pipeline.

[0074] Once the selected fixed-function unit has been notified, the control unit may write the data for performing the operation into the input buffer of the selected fixed-function unit (block 814). The control unit may then determine whether another operation should be scheduled (conditional block 816). In a particular implementation, one or more other operations may be scheduled by the scheduler such that each operation may be scheduled for another fixed-function unit of the pipeline (sequentially or non-sequentially).

[0075] In the case where it is determined that no other operations are to be scheduled (the "no" branch of conditional block 816), the control unit may identify the memory location to which the data associated with the output of the execution needs to be sent (block 818). Based on such identification, once the selected fixed-function unit notifies the control unit that the data is ready for use, the control unit may read the data from the output buffer of the selected fixed-function unit and transfer the data to the identified memory location. The control unit may then monitor the status of the fixed-function unit and transmit this information to the scheduler (block 820).

[0076] However, if the control unit determines based on the instructions received from the scheduler that other operations are to be scheduled (the "yes" branch of conditional block 816), the control unit may select another fixed-function unit and send the data of the execution result of the operation to the input buffer of the other selected fixed-function unit (block 822). Then, the control may monitor the status of the fixed-function unit again (block 820). In a particular implementation, the scheduler may access the workload status, error conditions, data flow, etc. of the fixed-function unit through the control unit and make scheduling decisions based on this accessed information.

[0077] It should be emphasized that the above particular implementations are merely non-limiting examples of particular implementations. Once the above disclosure is fully understood, many variations and modifications will become apparent to those skilled in the art. It is intended that the following claims be interpreted to cover all such variations and modifications.

Claims

1. A processor, the processor comprising: A pipeline, the pipeline including one or more fixed-function units; And A scheduler, the scheduler being configured to: In response to a first operating mode, schedule each of the operations by scheduling the operations with a first fixed-function unit of the pipeline for execution by any one of the one or more fixed-function units of the pipeline; And In response to a second operating mode, schedule the given operation for execution by a selected fixed-function unit of the pipeline by directly scheduling the given operation to a selected fixed-function unit other than the first fixed-function unit.

2. The processor according to claim 1, wherein the scheduler is configured to generate a scheduling graph including one or more nodes, and schedule each of the one or more nodes of the scheduling graph for execution by one or more of the fixed-function units.

3. The processor according to claim 1, wherein in response to the second operating mode, data generated by the selected fixed-function unit is not transmitted to the next fixed-function unit of the pipeline.

4. The processor according to claim 2, wherein the scheduler is configured to directly schedule two nodes of the scheduling graph to two different fixed-function units for concurrent execution.

5. The processor according to claim 1, wherein in response to the second operating mode, the scheduler is configured to directly read data related to the output of the second operation from an output buffer of the selected fixed-function unit.

6. The processor according to claim 5, wherein the scheduler is configured to transfer the data to a memory location, where the memory location is one of a memory subsystem associated with the processor or a mailbox accessible by the scheduler.

7. The processor according to claim 1, wherein the selected fixed-function unit is configured to send a notification to the scheduler, the notification indicating completion of execution of the second operation.

8. A method, the method comprising: In response to a first operating mode, schedule each of the operations by scheduling the operations with a first fixed-function unit of the pipeline for execution by any one of the one or more fixed-function units of the pipeline; And In response to a second operating mode, schedule the given operation for execution by a selected fixed-function unit of the pipeline by directly scheduling the given operation to a selected fixed-function unit other than the first fixed-function unit.

9. The method according to claim 8, the method further comprising: Generating, by the scheduler, a scheduling graph including one or more nodes; And Scheduling, by the scheduler, each of the one or more nodes of the scheduling graph for execution by one or more of the fixed-function units.

10. The method according to claim 9, wherein in response to the second operating mode, data generated by the selected fixed-function unit is not transmitted to the next fixed-function unit of the pipeline.

11. The method according to claim 9, wherein the method further comprises the scheduler directly scheduling two nodes of the scheduling graph to two different fixed function units for concurrent execution.

12. The method according to claim 9, wherein in response to the second operation mode, the scheduler directly reads data related to the output of the second operation from the output buffer of the selected fixed function unit.

13. The method according to claim 12, wherein the method further comprises the scheduler transferring the data to a memory location, where the memory location is one of the memory subsystems associated with a mailbox accessible by the scheduler.

14. The method according to claim 9, wherein the method further comprises the scheduler receiving a notification from the selected fixed function unit, the notification indicating completion of the execution of the second operation.

15. A computing system, the computing system comprising: a central processing unit; a graphics processing unit, the graphics processing unit comprising: a pipeline, the pipeline comprising one or more fixed function units; and a scheduler, the scheduler being configured to: in response to a first operation mode, schedule each of the operations in the pipeline by scheduling with a first fixed function unit of the pipeline for execution by any one of the one or more fixed function units of the pipeline; and in response to a second operation mode, schedule the given operation for execution by a selected fixed function unit of the pipeline by directly scheduling the given operation to a selected fixed function unit other than the first fixed function unit.

16. The computing system according to claim 15, wherein the scheduler is configured to generate a scheduling graph comprising one or more nodes, and schedule each of the one or more nodes of the scheduling graph for execution by one or more of the fixed function units.

17. The computing system according to claim 15, wherein in response to the second operation mode, data generated by the selected fixed function unit is not transferred to the next fixed function unit of the pipeline.

18. The computing system according to claim 15, wherein the scheduler is configured to directly schedule two nodes of the scheduling graph to two different fixed function units for concurrent execution.

19. The computing system according to claim 15, wherein in response to the second operation mode, the scheduler is configured to directly read data related to the output of the second operation from the output buffer of the selected fixed function unit.

20. The computing system according to claim 19, wherein the scheduler is configured to transfer the data to a memory location, where the memory location is one of the memory subsystems associated with a mailbox accessible by the scheduler.

Images