Executing a tensor instruction

Abstract

The present disclosure relates to performing tensor instructions, and specifically discloses apparatuses, systems, and techniques for performing operations asynchronously in a processor. In at least one embodiment, the processor executes at least one tensor instruction concurrently with one or more other instructions based at least in part on one or more indicators that the tensor instruction is asynchronous.

Description

Technical Field

[0001] Apparatus, systems, and methods for performing tensor operations (e.g., matrix multiplication-accumulation operations) in a processor. In at least one embodiment, one or more circuits are used to execute the at least one tensor instruction concurrently with one or more other instructions, based at least in part on one or more indicators that the at least one tensor instruction is asynchronous. Background Technology

[0002] When a thread executing a task (such as matrix multiplication) using a processing core is running by a streaming multiprocessor (SM) in a processor, the thread waits until the task is complete before performing other tasks. This can lead to a degraded software performance. Attached Figure Description

[0003] Figure 1 A system for executing an asynchronous instruction sequence according to at least one embodiment is shown;

[0004] Figure 2 A processor for executing an asynchronous instruction sequence according to at least one embodiment is shown;

[0005] Figure 3 An asynchronous instruction sequence in a processing queue according to at least one embodiment is shown;

[0006] Figure 4 An asynchronous instruction sequence in a processing queue according to at least one embodiment is shown;

[0007] Figure 5 A process for executing an asynchronous instruction sequence according to at least one embodiment is illustrated;

[0008] Figure 6 A process for executing an asynchronous instruction sequence for reusable operands according to at least one embodiment is illustrated;

[0009] Figure 7 A block diagram is shown of a driver and / or runtime comprising one or more libraries for providing one or more application programming interfaces (APIs) according to at least one embodiment;

[0010] Figure 8 An example data center system according to at least one embodiment is shown;

[0011] Figure 9 A system-on-a-chip (SOC) according to at least one embodiment is shown;

[0012] Figure 10A A parallel processor according to at least one embodiment is shown;

[0013] Figure 10B A processing cluster according to at least one embodiment is shown;

[0014] Figure 10C A graphics multiprocessor according to at least one embodiment is shown;

[0015] Figure 11 An accelerator processor according to at least one embodiment is shown;

[0016] Figure 12A A central processing unit according to at least one embodiment is shown;

[0017] Figure 12B The illustration shows an embodiment according to at least one of the embodiments. Figure 12A The core of the central processing unit;

[0018] Figure 13 Another accelerator processor according to at least one embodiment is shown;

[0019] Figure 14 A neuromorphic processor according to at least one embodiment is shown;

[0020] Figure 15 A supercomputer according to at least one embodiment is shown;

[0021] Figure 16 Another accelerator processor according to at least one embodiment is shown;

[0022] Figure 17 Another processor according to at least one embodiment is shown;

[0023] Figure 18 Another accelerator processor according to at least one embodiment is shown;

[0024] Figure 19 A tensor processing unit according to at least one embodiment is shown;

[0025] Figure 20 A RISC-V compatible processor according to at least one embodiment is shown;

[0026] Figure 21A and Figure 21B A language processing unit according to at least one embodiment is shown;

[0027] Figure 22 A software stack of a programming platform according to at least one embodiment is shown;

[0028] Figure 23 Software supported by a programming platform according to at least one embodiment is shown;

[0029] Figure 24A method for using at least one embodiment is shown. Figure 23 Compiled code executed on the programming platform;

[0030] Figure 25 An example of an autonomous vehicle and its system architecture according to at least one embodiment is shown;

[0031] Figure 26A The inference and / or training logic according to at least one embodiment is illustrated;

[0032] Figure 26B The inference and / or training logic according to at least one embodiment is shown; and

[0033] Figure 26C Training and deployment of a neural network according to at least one embodiment are illustrated. Detailed Implementation

[0034] Numerous specific details are set forth in the following description to provide a more thorough understanding of at least one embodiment. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0035] In at least one embodiment, the computing system uses processor execution threads that instruct the processor's circuitry to perform operations, such as tensor operations, asynchronously according to a processing queue. In at least one embodiment, the processor is part of and / or a combination of a graphics processing unit (GPU), a general-purpose GPU (GPGPU), a parallel processing unit (PPU), a central processing unit (CPU), a data processing unit (DPU), a system-on-a-chip (SoC), and others. In at least one embodiment, the processor includes multiple processing units (e.g., processor cores, tensor cores, computation units, execution units, or streaming multiprocessors), each of which executes or invokes a thread or bundle of threads to perform, compute, or implement tensor operations, such as matrix multiplication-accumulation operations.

[0036] In at least one embodiment, a thread or group of threads of the processing unit submits instructions to be stored in an operation processing queue. In at least one embodiment, the processing unit performs tensor operations, such as matrix multiplication-accumulation (MMA) operations, in the order indicated in the operation processing queue, rather than directly responding to instructions received by a thread (e.g., asynchronously). In at least one embodiment, tensor instructions for performing these tensor operations include indicators, modifiers, or parameters indicating that the tensor operations will be performed asynchronously. In at least one embodiment, the processing unit performs tensor operations indicated by a thread, independent of the thread executing other instructions. In at least one embodiment, the processing unit is performing an operation according to the next instruction in the queue while a thread is concurrently submitting an additional, different instruction to the operation processing queue. In at least one embodiment, if an instruction in the queue requires the output of an operation caused by another thread, the instruction uses a barrier to wait until the corresponding thread is notified that the operation has completed.

[0037] In at least one embodiment, a processing thread may submit instructions to a processing unit, which will notify the submitting thread, a different thread, or a group of threads when the processing unit has completed or executed a previously submitted instruction. In at least one embodiment, a processing thread may execute a first instruction to await indication that the output of a previously completed operation or instruction is available for access, reading, and / or modification.

[0038] In at least one embodiment, the data structure required for a given tensor operation or the memory location allocated for that operation (e.g., operands for a matrix multiplication-accumulation operation) is ready before the operation can be executed. In at least one embodiment, modification or otherwise access to the data structure or memory allocation is not permitted until after the tensor operation is complete. In at least one embodiment, the processor or a processing unit of the processor indicates when a given instruction or set of instructions has been executed or otherwise completed, indicating that the data structure or memory allocation used by the instruction can be accessed / modified. In one embodiment, the indication is a special asynchronous operation synchronized with a thread waiting to receive the indication. In at least one embodiment, the thread is a different thread from the thread providing the instructions to perform the operation. In at least one embodiment, the thread is the same thread as the thread providing the instructions.

[0039] In at least one embodiment, tensor operations (e.g., MMA) of instructions in the processing queue require operands stored in a memory location. In at least one embodiment, this memory location is part of at least one shared memory or dedicated tensor memory of a streaming multiprocessor. In at least one embodiment, the operands stored in the memory location can be reused for multiple operations indicated by a thread. In at least one embodiment, the thread may include instructions as parameters or additional metadata indicating that the operands will be reused in subsequent operations. In at least one embodiment, the operands are retained in the memory location until the subsequent operations are executed from the processing queue.

[0040] In at least one embodiment, various instructions submitted by a thread are executed in stages. In at least one embodiment, a first stage is executed by a first portion of a processing unit (e.g., a streaming multiprocessor) and governed by a first memory barrier; while a second stage is executed by a second portion of the processing unit and governed by a second memory barrier. In at least one embodiment, while the first portion executes the operations of the first stage, operands for the second stage are loaded, modified, or written to the second portion. In at least one embodiment, while the second portion executes the operations of the second stage, operands for the first stage are loaded, modified, or written to the first portion. In at least one embodiment, when the operations of the first stage are completed, the first memory barrier issues an indication that the operands for the first stage are ready to be modified. In at least one embodiment, when the operations of the second stage are completed, the second memory barrier issues an indication that the operands for the second stage are ready to be modified.

[0041] In at least one embodiment, when one or more threads or thread bundles of the processor execute instructions, the processor stores operands (e.g., input matrices) and / or accumulated results in memory. In at least one embodiment, the memory includes a register file, but the instructions require a set of threads (or a thread bundle) to lock the registers of the register file so that no other operation can store information in those registers. In at least one embodiment, the memory includes shared memory (SMEM) and / or tensor memory (TMEM), which can release the register file for use by other operations even while the MMA is being executed. In at least one embodiment, by using a processing queue, the other operations can be scheduled and executed by a portion of the streaming multiprocessor while the MMA is being executed by another portion of the streaming multiprocessor. In at least one embodiment, since a single thread can add instructions to the queue, other threads of the streaming multiprocessor can be used for other operations.

[0042] Figure 1An example of a system 100 according to at least one embodiment is shown. The system 100 may include software and hardware for: executing a tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or executing one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are in program order preceding one or more pause instructions; adding one or more instructions to one or more memory queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicating to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or executing one or more instructions in an order independent of the order in which the instructions were submitted by at least one of a graphics processing unit (GPU) processing core or an MMA accelerator; or otherwise performing any of the operations described herein. System 100 may include a storage device 102 and a processor 108. Storage device 102 may include, for example, memory, cache, or other storage devices that will be further described herein. Storage device 102 may be separate from processor 108, or storage device 102 may be included in processor 108 (e.g., in storage device 112). In at least one embodiment, software program 104 and / or software library (or instructions) 106 may be stored in memory, cache, or other storage devices and provided to processor 108 to cause one or more circuits of processor 108 to perform the operations described herein. In at least one embodiment, software program 104 and / or software library (or instructions) 106 may be integrated into one or more circuits of processor 108. Software program 104, which can be used to perform any of the operations described herein, may be stored in storage device 102. In at least one embodiment, software program 104 may include one or more software modules. In at least one embodiment, these software modules include software configured to: execute the tensor instruction concurrently with one or more other instructions, based at least in part on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more memory queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted by at least one of a processing core of a graphics processing unit (GPU) or an MMA accelerator.

[0043] In at least one embodiment, unless the context explicitly specifies otherwise, as used in any implementation described herein, a module refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. In at least one embodiment, software is embodied as a software package, code, and / or instruction set or instructions, while “hardware” as used in any implementation described herein includes, individually or in any combination, for example, hardwired circuitry, programmable circuitry, state machine circuitry, fixed-function circuitry, execution unit circuitry, and / or firmware storing instructions executed by programmable circuitry. In at least one embodiment, a module may be embodied collectively or individually as circuitry forming part of a larger system (e.g., an integrated circuit (IC), a system-on-a-chip (SoC), etc.). In at least one embodiment, a module performs one or more processes in conjunction with any suitable processing unit and / or combination of processing units (such as one or more CPUs, GPUs, GPGPUs, PPUs, and / or variants thereof, including those further described herein).

[0044] In at least one embodiment, software program 104 may include a collection of software code, commands, instructions, or other text sequences for instructing a computing device to perform one or more computational operations and / or invoke one or more sets of other instructions (such as APIs or API functions or instruction set architecture (ISA) level instructions) to be run or otherwise executed. Instructions (e.g., hardware instructions) or microcode may include ISA level instructions, which may include native ISA instructions or non-native ISA instructions. Software program 104 and / or software library (or instruction) 106 (e.g., one or more modules) may be distributed among multiple processors that communicate via buses, networks, writes to shared memory, and / or any suitable communication method (such as the communication methods described herein).

[0045] In at least one embodiment, system 100 may include one or more software libraries 106, which may provide one or more API and / or ISA instructions. In at least one embodiment, one or more API and / or ISA instructions may be used to: execute a tensor instruction concurrently with one or more other instructions, at least in part based on the fact that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein the one or more first instructions are programmatically ordered before one or more pause instructions; add one or more instructions to one or more storage queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator. In at least one embodiment, one or more software libraries 106 may be included in a driver and / or runtime environment. In at least one embodiment, software library 106 (e.g., including one or more API and / or ISA instructions) may include software instruction sets that, when executed or otherwise implemented, cause processor 108 to perform one or more computational operations, such as any operations described herein. In at least one embodiment, one or more API and / or ISA instructions may be distributed or otherwise provided as part of one or more packages of software library 106, runtime, drivers, and / or other software and / or executable code further described herein. In at least one embodiment, one or more API and / or ISA instructions may perform one or more computational operations in response to invocation of software program 104.

[0046] Processor 108 may include any number of processors and any suitable processing units and / or combinations of processing units, such as, but not limited to, a central processing unit (“CPU”), a graphics processing unit (“GPU”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, parallel processors, GPGPUs, DPUs, and / or variations thereof, including those further described herein), including any processors described herein, such as, but not limited to, those described herein. Figure 9-21BThe processor 108 can retrieve or fetch instructions (e.g., one or more API and / or ISA instructions) from storage device 102 using, for example, instruction fetch 116 (e.g., for an instruction fetch phase). Instructions may include: instructions for executing tensor instructions concurrently with one or more other instructions, based at least in part on one or more indicators that at least one tensor instruction is asynchronous; or instructions for executing one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are programmatically preceding one or more pause instructions; instructions for adding one or more instructions to one or more storage queues by a single thread of a cooperative thread array (CTA) of one or more SMs; instructions to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or instructions for executing one or more instructions in an order independent of the order in which the instructions were submitted by at least one of the processing cores or MMA accelerators of a graphics processing unit (GPU). In at least one embodiment, processor 108 may include a storage device 112 and an instruction queue 110 for storing and queuing instructions fetched from storage device 102. In at least one embodiment, the fetched instructions may be decoded by decoder 118 to determine what operation processor 108 should perform (e.g., during the instruction decoding phase). In at least one embodiment, processor 108 may fetch additional operands (data) available for the instructions, and these operands may be stored, for example, in registers or storage device 112. In at least one embodiment, micro-operations 120 may perform operations on data stored in one or more registers or storage devices 112. For example, each step of the instructions fetched by processor 108 may be broken down during execution, so processor 108 may execute instructions step-by-step through a series of micro-operations 120. In at least one embodiment, program counter (PC) 114 may hold an address for the next instruction and may be updated to point to the next instruction to be executed by processor 108.

[0047] In at least one embodiment, processor 108 may execute instructions (e.g., during execution phase). For example, processor 108 may perform tensor operations specified by the instructions, such as arithmetic operations, logical operations, or data transfers. In at least one embodiment, computing unit 122 may execute instructions to perform any of the operations described herein. In at least one embodiment, computing unit may include ALU 124 (Arithmetic Logic Unit), which can be used to perform arithmetic and logical operations. In at least one embodiment, computing unit may include FPU (Floating Point Unit) 126, which can be used to perform floating-point operations. In at least one embodiment, other circuitry 128 may be used to perform other operations, such as vector operations and / or scalar operations. In at least one embodiment, accelerator 130 may include one or more matrix multiplication accelerators, one or more parallel processing units (PPUs) (such as GPUs), or any other accelerators or processors further described herein. In at least one embodiment, software program 104 may utilize one or more API and / or ISA instructions to perform various computational operations, such as matrix multiplication, arithmetic operations, or any other computational operations further described herein, using accelerator 130. In at least one embodiment, one or more computational operations using accelerator 130 may include at least one or more sets of computational operations that will be accelerated by execution at least partially by accelerator 130, including: executing the tensor instruction concurrently with one or more other instructions based at least partially on one or more indicators that at least one tensor instruction is asynchronous; or executing one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are in program order preceding one or more pause instructions; adding one or more instructions to one or more memory queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicating to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or executing one or more instructions in an order independent of the order in which the instructions were submitted by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator.

[0048] In at least one embodiment, system 100 can be used to execute one or more instructions, including functions or operations, such as combining... Figure 2-7The functions or operations described herein. In at least one embodiment, system 100, including one or more processors, causes one or more circuits to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that the tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are in program order preceding one or more pause instructions; add one or more instructions to one or more memory queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted and / or otherwise perform the operations described herein. In at least one embodiment, system 100 is included Figure 2-7 The system shown and / or otherwise includes Figure 2-7 The system shown is configured such that one or more circuits: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions; add one or more instructions to one or more memory queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted and / or otherwise perform the operations described herein. In at least one embodiment, system 100 includes Figure 8-26COne or more pieces of hardware, such as those shown, are used for: executing the tensor instruction concurrently with one or more other instructions, based at least in part on one or more indicators that at least one tensor instruction is asynchronous; or executing one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are in program order preceding one or more pause instructions; adding one or more instructions to one or more memory queues by a single thread of a cooperative thread array (CTA) of one or more SMs; indicating to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or executing one or more instructions in an order independent of the order in which the instructions were submitted and / or otherwise performing the operations described herein.

[0049] Figure 2 A processor for executing asynchronous instruction sequences according to at least one embodiment is illustrated. In at least one embodiment, processor 200 is part of and / or a combination of a graphics processing unit (GPU), a general-purpose GPU (GPGPU), a parallel processing unit (PPU), a central processing unit (CPU), a data processing unit (DPU), a system-on-a-chip (SoC), and so on. In at least one embodiment, the processor includes any other type of processor further described herein. In at least one embodiment, processor 200 includes: one or more processor cores, including one or more streaming multiprocessors (SMs) 210; and a global memory 280 storing information for use by the SMs 210. In at least one embodiment, processor 200 corresponds to processor 108.

[0050] In at least one embodiment, each streaming multiprocessor (SM) 210 includes one or more MMA accelerator units 220 (e.g., tensor cores) that perform arithmetic or logical computations as instructed by tensor instructions submitted by threads in a cooperative thread array (CTA) 260. In at least one embodiment, the threads submit these tensor instructions to an operation processing queue 270 that stores the instructions until the accelerator unit 220 or other processing unit becomes available to perform operations according to the instructions. In at least one embodiment, any thread of the CTA 260 may submit instructions to the operation processing queue 270, wherein the accelerator unit 220 may execute any instructions in the operation processing queue 270 in any independent order. In at least one embodiment, the tensor instructions for performing these tensor operations include indicator or modifier or parameter indicating whether the tensor operation will be performed asynchronously or independently.

[0051] In at least one embodiment, instructions in operation processing queue 270 cause a processing unit (e.g., accelerator unit 220) to perform an operation using one or more operands. In at least one embodiment, accelerator unit 220 is associated with various memory elements of SM 210 that store operands to be used for the arithmetic or logical computation. In at least one embodiment, these various memory elements include shared memory 230 and / or accelerator memory 240. In at least one embodiment, performing MMA using accelerator unit 220 includes multiplying each element of a row in a first input matrix, or a first operand, with each element of a column in a second input matrix, or a second operand, and accumulating the result of the multiplication operation. In at least one embodiment, the first operand is stored in accelerator memory 240 and / or shared memory 230, while the second operand is stored in shared memory 230. In at least one embodiment, the accumulated result is stored in accelerator memory 240. In at least one embodiment, accelerator unit 220 is a processor core or accelerator circuit configured to perform matrix arithmetic (such as matrix multiplication) or deep learning matrix operations (such as convolution operations for neural network training and inference). In at least one embodiment, a processor (such as a CPU) provides instructions to a GPU to perform arithmetic operations using tensor cores provided on the GPU. In at least one embodiment, each tensor core operates on an input matrix and performs a matrix multiplication-accumulation operation (D = A × B + C), where A, B, and C are operand matrices, and D is the accumulated result matrix.

[0052] In at least one embodiment, the streaming multiprocessor (SM) 210 uses a thread of the CTA 260 to submit instructions to perform an operation (such as MMA). In at least one embodiment, the instructions are stored in an operation processing queue 270. In at least one embodiment, when the accelerator unit 220 becomes available for operation, the next instruction is retrieved from the operation processing queue 270 and executed accordingly. In at least one embodiment, after executing a predetermined number of instructions or operations, the accelerator unit 220 indicates to waiting threads that the predetermined number of instructions has been completed.

[0053] In at least one embodiment, SM 210 stores information related to the MMA operation (such as operands or accumulated results) in memory. In at least one embodiment, the instruction causes SM 210 to retrieve at least a portion of the operands from global memory 280 and store said portion using shared memory 230 and / or accelerator memory 240 for MMA computation. In at least one embodiment, the instruction causes SM 210 to output the MMA computation result from accelerator unit 220, store said output using shared memory and / or accelerator memory 240, and accumulate said output to the previously stored computation result. In at least one embodiment, SM 210 also includes register 250 for storing data retrieved from global memory 280 or shared memory 230 and / or instructions provided from CTA 260. In at least one embodiment, register 250 is not used to store information necessary for performing the MMA, but can be used to perform other arithmetic or logical computations, such as those instructed by different threads from CTA 260.

[0054] In at least one embodiment, some or all of the processes described herein (or any other described processes, or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions and / or other data, and may be implemented as executable instructions that execute jointly on processor 200 or a combination of processors 200. In at least one embodiment, the executable instructions and / or other data may be stored on a non-transitory computer-readable storage medium (e.g., a computer program persistently stored on a magnetic, optical, or flash memory medium).

[0055] In at least one embodiment, the processor 200 can execute references Figure 3-6 The process described can be executed as referenced. Figure 7 The instructions or application programming interface (API) functions described herein may be used by any suitable system, such as references Figure 8-26C The or execution Figure 8-26C A computing device for the process. In at least one embodiment, Figure 2 The logic and hardware architecture can be integrated into the execution. Figure 1 and Figure 3-7 The processes disclosed in the system, processor, and architecture. For example, Figure 2 The logic / hardware structure within can execute at least part or all of processes or APIs 300, 400, 500, 600, and / or 700. In at least one embodiment, Figure 2The disclosed system or apparatus includes one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous. In at least one embodiment, Figure 2 The systems or apparatus disclosed herein cause a processor to use one or more memory queues to store one or more instructions, which will be executed by one or more portions of one or more streaming multiprocessors (SMs) until said one or more portions become available to execute said one or more instructions. In at least one embodiment, Figure 2 The disclosed system or apparatus adds one or more instructions to one or more storage queues via a single thread in a cooperative thread array (CTA) of one or more SMs. In at least one embodiment, Figure 2 The systems or apparatus disclosed herein cause a processor to indicate that one or more matrix multiplication-accumulation (MMA) operations have been completed. In at least one embodiment, Figure 2 The system or apparatus disclosed herein executes one or more instructions by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order indicated by one or more storage queues.

[0056] Figure 3 An asynchronous instruction sequence 300 in a processing queue according to at least one embodiment is shown. In at least one embodiment, the processor (e.g., Figure 1 The processor 108 or Figure 2 The processor 200 in the CTA executes the instruction sequence 300, utilizing a thread in the CTA to send data to the processing queue (e.g., ...). Figure 2 The operation processing queue 270 provides instructions that a computing unit (e.g., accelerator unit 220) can execute when it is ready. In at least one embodiment, the processor (e.g., processor 108 or processor 200) stores at least one tensor instruction and one or more other instructions by executing the instruction sequence 400, which will be executed concurrently at least in part based on the tensor instruction being an asynchronous indicator.

[0057] In at least one embodiment, CTA (e.g., Figure 2 The first thread in CTA 260 (by...) Figure 3 (As shown in the left column of the table) Submits a set of tensor instructions 302 that will perform a first MMA operation (MMA0) and then perform other MMA operations (MMA1-MMA3). In at least one embodiment, this set of instructions 302 is stored as a storage instruction 312 in a processing queue (e.g., Figure 2 The operation processing queue is 270, and is handled by... Figure 3In at least one embodiment, a computational unit (such as accelerator unit 220 or tensor core) executes the stored instructions 312 in the order in which the instructions are stored (e.g., MMA0 first, then MMA1, etc.). In at least one embodiment, a computational unit (such as accelerator unit 220 or tensor core) executes the stored instructions 312 in a non-sequential order (e.g., MMA1 first, then MMA0, etc.). In at least one embodiment, the tensor instructions for performing these tensor operations include an indicator, modifier, or parameter (e.g., mma.async) indicating that the tensor operations will be performed asynchronously or in a non-sequential order.

[0058] In at least one embodiment, the instructions for performing the MMA operation require that any operand matrices or accumulation result matrices to be used in the MMA operation be stored and / or allocated in memory before the instructions can be submitted to the processing queue. In at least one embodiment, once these operands or accumulation results have been loaded into memory, the operands cannot be accessed, read, written to, or otherwise modified by a user or other external program until the instructions and / or operation are completed. In at least one embodiment, the processing unit of the SM (such as an accelerator unit) provides an indication that an operation or group of operations has been completed, indicating that previously unavailable operands are available for access, reading, and / or modification. In at least one embodiment, upon receiving the indication, the accumulation result matrix can be read out, thereby providing the results of the MMA operation to a user or software program or module that requires these accumulation results.

[0059] In at least one embodiment, a predetermined number of instructions to be executed is set by a programmer or another processor before the availability indication is returned. In at least one embodiment, the predetermined number of instructions is determined by the maximum amount of cache memory available to store the instructions. In at least one embodiment, the predetermined number of instructions is set by a user through a software program that executes the instructions or operations. In at least one embodiment, after the predetermined number of instructions is submitted, a barrier arrival instruction 304 is submitted to the processing queue. In at least one embodiment, after a processing unit (e.g., accelerator unit 220) executes the stored instruction 312 and performs all indicated operations, the processing unit receives the stored barrier arrival instruction 314 and provides the barrier arrival indication to the thread designated to wait for the barrier arrival indication 324, indicating that the indicated operation has been completed. In at least one embodiment, the waiting thread is a different thread in the CTA (e.g., such as...). Figure 3 (as indicated in the right column), rather than the thread used to submit instructions to the processing queue (e.g., as indicated in the right column). Figure 3(As indicated in the left column). In at least one embodiment, the waiting thread is the same thread as the thread used to submit the instruction (e.g., as per [reference]). Figure 4 (As shown). In at least one embodiment, after the waiting thread receives an indication that the previously indicated operation has been completed, the waiting thread performs any or all necessary closing operations of the MMA required by the application executing the MMA, such as accumulating partial results, bias addition, quantization, normalization, applying an activation function, or transferring the results to another storage location.

[0060] In at least one embodiment, a second set of instructions 306, indicating that a second set of MMA operations (MMA4-MMA7) will be executed, is submitted to a processing queue. In at least one embodiment, these second store instructions 306 are executed by a processing unit (e.g., accelerator unit 220) in a sequential or non-sequential order. In at least one embodiment, since these instructions are received asynchronously, these second store instructions 306 are executed concurrently with the time when the thread designated for wait 324 receives an indication that previously unavailable operands (e.g., operands MMA0-MMA3) or accumulated results are available for access, reading, and / or modification, and performs a closing operation on these newly available operands or results.

[0061] In at least one embodiment, after the second set of instructions is submitted, a second barrier arrival instruction 308 is also submitted. In at least one embodiment, after a processing unit (e.g., accelerator unit 220) executes the second storage instruction 316 and performs all operations as instructed, the processing unit receives the stored barrier arrival instruction 318 and provides this second instruction to the thread previously designated to wait for the completion of the operation 328, so as to perform a closing operation or other necessary operations on the operands or results of the operation.

[0062] In at least one embodiment, a computing device (such as system 100, processor 200, or any suitable system, e.g., reference) Figure 8-26C The described computing device can execute instruction sequence 300. In at least one embodiment, by executing instruction sequence 300, the processor can at least partially execute the reference... Figure 4-6 The described process, or a reference that can be executed. Figure 7 The described application programming interface (API) functions, or Figure 8-26C Other processes. In at least one embodiment, by performing Figure 3 The instruction sequence 300 allows the processor to use one or more memory queues to store one or more instructions to be executed by one or more portions of one or more streaming multiprocessors (SMs) until said one or more portions become available for execution of said one or more instructions. In at least one embodiment, by executing Figure 3 The processor can, using instruction sequence 300,: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein the one or more first instructions are scheduled before one or more pause instructions in program order. In at least one embodiment, by executing Figure 3 The instruction sequence 300 allows the processor to add one or more instructions to one or more memory queues via a single thread in a cooperative thread array (CTA) of one or more SMs. In at least one embodiment, by executing Figure 3 The processor can indicate, via instruction sequence 300, that one or more matrix multiplication-accumulation (MMA) operations have been completed. In at least one embodiment, by executing... Figure 3 The instruction sequence 300, at least one of the processing cores of the graphics processing unit (GPU) or the MMA accelerator, can execute thread-submitted instructions in the order indicated by one or more storage queues.

[0063] Figure 4 An asynchronous instruction sequence 400 in a processing queue according to at least one embodiment is shown. In at least one embodiment, the processor (e.g., Figure 1 The processor 108 or Figure 2 The processor 200 in the process executes the tensor instruction sequence 400, utilizing the CTA thread to send data to the processing queue (e.g., ...). Figure 2 The operation processing queue 270 provides instructions that the computing unit (e.g., accelerator unit 220) can execute when it is ready. In at least one embodiment, the processor (e.g., processor 108 or processor 200) executes the tensor instruction sequence 400 concurrently with one or more other instructions, based at least in part on one or more indicators that at least one tensor instruction is asynchronous. In at least one embodiment, by executing the instruction sequence 400, the thread that submits the instructions to the CTA in the processing queue is the same thread that is waiting for notification of the completion of one or more operations, in order to identify when operands or accumulated results of the MMA can be accessed, read, and / or modified. In at least one embodiment, by executing the instruction sequence 400, the thread further causes a closing operation or other necessary operation to be executed when it receives the notification.

[0064] In at least one embodiment, the tensor instruction sequence 400 is divided into multiple stages. In at least one embodiment, the instructions are divided into two stages, wherein instructions corresponding to a first tensor operation are grouped into a first stage with a first memory barrier, and instructions corresponding to a second tensor operation are grouped into a second stage with a second memory barrier. In at least one embodiment, the number of instruction divisions and / or the number of operations grouped into a single stage depends on the maximum amount of cache memory available to store all instructions. In at least one embodiment, if the cache can store four instructions, two instructions can be allocated to the first stage and two instructions to the second stage, such that all these instructions can be stored in a processing queue and executed asynchronously. In at least one embodiment, the tensor instructions for performing these tensor operations include an indicator, modifier, or parameter indicating that the tensor operation will be executed asynchronously.

[0065] In at least one embodiment, a cooperative thread array (e.g., Figure 2 A single thread in CTA 260 causes the SM (e.g., SM 210) to allocate and load the operand matrix and the accumulated result matrix into the memory of the first stage of the matrix multiplication-accumulation operation (e.g., MMA0 and MMA1). Figure 2 The shared memory 230 and / or accelerator memory 240 are used to update the operand matrix and accumulation result matrix in these memories. In at least one embodiment, after the MMA operands are loaded, the single thread submits these first-stage instructions to execute the MMA that will be stored in the processing queue.

[0066] In at least one embodiment, when the processing unit is available, the processing unit or computing unit (e.g., accelerator unit 220) retrieves the first-stage storage instruction 412. In at least one embodiment, the processing unit then executes the first-stage storage instruction 412 and performs various operations corresponding to those instructions.

[0067] In at least one embodiment, the processing unit executes a predetermined number of instructions, wherein the predetermined number is determined based on the maximum amount of cache memory available to store the instructions, or based on a value preset by a programmer or user, which has been encoded into the software program executing the instructions. In at least one embodiment, after a thread submitting instructions to a processing queue has submitted the predetermined number of instructions, the thread then additionally submits a barrier arrival instruction 404 to the processing queue. In at least one embodiment, when the processing unit (e.g., accelerator unit 220) retrieves the stored barrier arrival instruction 414, the processing unit provides an indication to the first barrier that the operands or results used by the first-stage MMA operation are accessible, readable, and / or writable. In at least one embodiment, if the operand matrix and result matrix are accessible, the processing unit also additionally performs an MMA termination operation. In at least one embodiment, after all termination operations have been performed, the processing unit then provides an indication to the waiting thread 410 that the first-stage instructions have been completed at the first barrier.

[0068] In at least one embodiment, when the processing unit performs the end operation for the first stage, the thread causes the SM (e.g., SM 210) to allocate and load the operand matrix and the accumulated result matrix into the memory of the second stage of the matrix multiplication-accumulation operation (e.g., MMA2 and MMA3). Figure 2 The operand matrix and accumulated result matrix in the shared memory 230 and / or accelerator memory 240 are updated. In at least one embodiment, after the second-stage MMA operands are loaded, the single thread submits these second-stage instructions to execute the MMA operation to be stored in the processing queue. In at least one embodiment, operands used in previous MMA operations can be reused to perform MMA operations in the second stage or subsequent stages, in which case the operand matrix and / or result matrix are passed to subsequent computations without any modification.

[0069] In at least one embodiment, when the processing unit (e.g., accelerator unit 220) is available (e.g., when the closing operation of the previous stage 414 is completed), the processing unit retrieves these second-stage store instructions 416. In at least one embodiment, the processing unit executes the second-stage store instructions 416 to perform all second-stage MMA operations using the stored operands. In at least one embodiment, after a predetermined number of instructions have been submitted to the processing queue, the thread then additionally submits a second-stage barrier arrival instruction 408 to the processing queue. In at least one embodiment, when the processing unit retrieves the stored second-stage barrier arrival instruction 418, the processing unit provides an indication to the second barrier that the operands or results used by the second-stage MMA operations are accessible, read, and / or written. In at least one embodiment, if the operand matrix and result matrix are accessible, the processing unit additionally performs the second-stage MMA closing operation.

[0070] In at least one embodiment, the single thread submitting the instruction waits to receive an indication that the first stage processing is complete after submitting the second-stage instruction to the processing queue. In at least one embodiment, the processing unit provides the waiting thread 410 with an indication that the first-stage instruction has been completed while performing the second-stage operation. In at least one embodiment, after receiving the indication, the thread continues to submit instructions to execute the first-stage operation while the second stage is being processed. In at least one embodiment, since the operation processing queue can be processed asynchronously, the two stages can operate concurrently, wherein while the first stage is performing a computation operation, the second stage is allocating or reclaiming operands; and while the second stage is performing a computation operation, the first stage is allocating or reclaiming operands.

[0071] In at least one embodiment, a computing device (such as system 100, processor 200, or any suitable system, e.g., reference) Figure 8-26C The described computing device can execute instruction sequence 400. In at least one embodiment, by executing instruction sequence 400, the processor can at least partially execute the reference... Figure 3 , Figure 5 and Figure 6 The described process, or a reference that can be executed. Figure 7 The described application programming interface (API) functions, or Figure 8-26COther processes. In at least one embodiment, by executing the instruction sequence 400, the processor (e.g., processor 108 or processor 200) executes the tensor instruction concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous, or asynchronously executes one or more second instructions in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are preceding one or more pause instructions in the program sequence. In at least one embodiment, by executing Figure 4 The instruction sequence 400 allows the processor to use one or more memory queues to store one or more instructions to be executed by one or more portions of one or more streaming multiprocessors (SMs) until said one or more portions become available for execution of said one or more instructions. In at least one embodiment, by executing Figure 4 The instruction sequence 400 allows the processor to add one or more instructions to one or more memory queues via a single thread in a cooperative thread array (CTA) of one or more SMs. In at least one embodiment, by executing Figure 4 The processor can indicate, via instruction sequence 400, that one or more matrix multiplication-accumulation (MMA) operations have been completed. In at least one embodiment, by executing... Figure 4 The instruction sequence 400, at least one of the processing cores of the graphics processing unit (GPU) or the MMA accelerator, can execute instructions submitted by the thread in the order indicated by one or more storage queues.

[0072] Figure 5 A process 500 for executing an asynchronous instruction sequence according to at least one embodiment is illustrated. In at least one embodiment, by executing process 500, a processor (such as...) Figure 1 processor 108 or Figure 2 The processor 200) uses threads in the CTA to send data to the processing queue (e.g., Figure 2The operation processing queue 270 provides instructions that the computing unit (e.g., accelerator unit 220) can execute when the computing unit is available. In at least one embodiment, by execution process 500, the processor (e.g., processor 108 or processor 200) executes the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous. In at least one embodiment, by execution process 500, the thread that submits the instruction to the CTA in the processing queue is the same thread that waits for notification of the completion of one or more operations to identify when operands or accumulated results of the MMA can be accessed, read, and / or modified. In at least one embodiment, by execution process 500, when the thread receives the notification, the thread additionally causes a closing operation or other necessary operation to be executed.

[0073] In at least one embodiment, in step 502, the processor allocates and / or loads the operand matrix and the accumulation result matrix into memory (e.g., Figure 2 (in shared memory 230 and / or accelerator memory 240). In at least one embodiment, in step 504, the processor then identifies the number of stages to be executed and / or the number of operands to be executed per stage. In at least one embodiment, the predetermined number of stages or instructions is determined by the maximum amount of cache memory available to store the tensor instructions, or is indicated in a user-designed software program. In at least one embodiment, after the number of stages, instructions, and / or operands are identified at step 504, process 500 continues concurrently to steps 506 and 516.

[0074] In at least one embodiment, in step 516, the processor designates a waiting thread to perform a wait operation to receive an indication that one or more operations have been completed. In at least one embodiment, the designated thread is the same thread that submitted the instruction to the operation processing queue. In at least one embodiment, the designated thread is a different thread from the thread that submitted the instruction to the operation processing queue, and is a thread specifically designated to wait for the indication.

[0075] In at least one embodiment, in step 506, the processor uses a thread to submit instructions corresponding to the operation to be performed and any stages identified in step 504. In at least one embodiment, these instructions are stored in an operation processing queue (e.g., Figure 2 In the operation processing queue 270), the processing unit can retrieve the stored instructions when the processor's processing unit (e.g., accelerator unit 220) is available.

[0076] In at least one embodiment, in step 508, the processor uses a thread to submit a barrier operation instruction to the operation processing queue. In at least one embodiment, the instruction is used to provide an indication or notification that an operation corresponding to a given stage has been completed.

[0077] In at least one embodiment, in step 510, the processor's processing unit retrieves stored instructions from the queue and executes the queued instructions. In at least one embodiment, these instructions cause the processor to perform a matrix multiplication-accumulation operation on operands loaded into memory at step 502 or step 520. In at least one embodiment, these instructions are executed in a sequence stored in the operation processing queue. In at least one embodiment, the instruction sequence is in the order submitted by the processor thread. In at least one embodiment, the instruction sequence is in an order different from the order submitted by the processor thread. In at least one embodiment, the tensor instruction sequence includes more than one instruction submitted by the processor thread, and the processing unit specifies the sequence in which the instructions will be executed. In at least one embodiment, the tensor instructions for performing these tensor operations include an indicator, modifier, or parameter indicating that the tensor operation will be executed asynchronously. In at least one embodiment, the queued instructions are executed concurrently with each other, and the instructions may complete at different times. In at least one embodiment, if one queued instruction completes while another instruction is still executing, a new instruction can be retrieved from the operation processing queue for concurrent execution.

[0078] In at least one embodiment, in step 512, the processor identifies whether the current processing stage is complete. In at least one embodiment, a stage is considered complete when the processor completes a predetermined number of instructions (such as the number of tensor instructions identified in step 504). In at least one embodiment, if additional instructions will be executed ("No" in step 512), the current stage is not considered complete, and the processor executes the next queued instruction. In at least one embodiment, if all predetermined instructions have been completed ("Yes" in step 512), process 500 continues to step 514.

[0079] In at least one embodiment, in step 514, once all predetermined instructions have completed, the processor performs a barrier arrival operation and indicates or notifies the waiting thread set in step 516 that the memory barrier has been reached. In at least one embodiment, the indication provides notification to other parts of the processor that the operation has been completed and allows access to, reading, and / or modification of any operand matrices and result matrices of these operations. In at least one embodiment, the indication also notifies the processor that the final operation of the matrix multiplication can be performed because the matrix is ​​accessible and modifiable.

[0080] In at least one embodiment, in step 520, the processor performs any necessary closing operations corresponding to the operations performed in the current stage. In at least one embodiment, the processor may additionally modify and load operands that will be used in subsequent operations, or it may reuse current operands so as not to reload the data when the same data will be used.

[0081] In at least one embodiment, in step 522, the processor then submits additional instructions to the operation processing queue. In at least one embodiment, these additional instructions correspond to the instructions for the next stage identified at step 504. In at least one embodiment, these additional instructions correspond to new operations to be processed by the processor, in which case new instructions and new stages may be determined. In at least one embodiment, once the additional instructions are submitted to the operation processing queue, the processing unit retrieves these instructions and executes the specified operation, repeating process 500 from step 510. In at least one embodiment, these queued additional instructions are executed concurrently with each other, and concurrently with other instructions queued in step 506 but not yet completed, and any of these instructions may complete at different times. In at least one embodiment, if a queued instruction completes while another instruction is still being executed, a new instruction may be retrieved from the operation processing queue for concurrent execution.

[0082] In at least one embodiment, process 500 may be performed by a computing device (such as system 100, processor 200, or any suitable system, such as a reference datum). Figure 8-26C The described computing device is used for execution. In at least one embodiment, through execution process 500, the processor can at least partially execute the reference. Figure 3 , Figure 4 and Figure 6 The described process, or a reference that can be executed. Figure 7 The described application programming interface (API) function, or execution Figure 8-26C Other processes in the process. In at least one embodiment, by executing process 500, the processor can execute the tensor instruction concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous, or asynchronously execute one or more second instructions in parallel with one or more first instructions to be executed by one or more first circuits, wherein the one or more first instructions are preceding one or more pause instructions in the program sequence. In at least one embodiment, by executing Figure 5In process 500, the processor may use one or more storage queues to store one or more instructions to be executed by one or more portions of one or more streaming multiprocessors (SMs) until said one or more portions become available to execute said one or more instructions. In at least one embodiment, by executing Figure 5 In process 500, the processor can add one or more instructions to one or more memory queues via a single thread in a cooperative thread array (CTA) of one or more SMs. In at least one embodiment, by executing Figure 5 In process 500, the processor can indicate that one or more matrix multiplication-accumulation (MMA) operations have been completed. In at least one embodiment, by executing Figure 5 In process 500, at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator can execute instructions submitted by a thread in the order indicated by one or more storage queues.

[0083] Figure 6 A process 600 for executing an asynchronous instruction sequence of reused operands according to at least one embodiment is illustrated. In at least one embodiment, by executing process 600, a processor (such as...) Figure 1 The processor 108 or Figure 2 The processor 200 in the CTA uses threads in the CTA to send data to the processing queue (e.g., processor 200). Figure 2 The operation processing queue 270 provides instructions that the computing unit (e.g., accelerator unit 220) can execute when the computing unit is available. In at least one embodiment, by execution process 600, the processor (e.g., processor 108 or processor 200) executes the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous. In at least one embodiment, by execution process 600, the thread that submits the instruction to the CTA in the processing queue is the same thread that is waiting for notification that one or more operations have been completed, thereby identifying when operands or accumulated results of the MMA can be accessed, read, and / or modified. In at least one embodiment, by execution process 600, when the thread receives the notification, the thread additionally causes a closing operation or other necessary operation to be executed.

[0084] In at least one embodiment, in step 602, the processor allocates and / or loads the operand matrix and the accumulation result matrix into memory (e.g., Figure 2(In shared memory 230 and / or accelerator memory 240). In at least one embodiment, as indicated by the maximum amount of cache memory available to store the instructions or as indicated in a user-designed software program, the processor additionally identifies the number of stages to be executed and / or the number of tensor operations to be performed per stage. In at least one embodiment, tensor instructions for performing these tensor operations include an indicator or modifier or parameter indicating that the tensor operations will be performed asynchronously.

[0085] In at least one embodiment, in step 604, the processor uses threads to submit instructions for performing operations to an operation processing queue (e.g., Figure 2 In the operation processing queue 270), the processing unit of the processor (e.g., accelerator unit 220) can retrieve the stored instructions when the processing unit is available.

[0086] In at least one embodiment, in step 606, the processor's processing unit (e.g., Figure 2 The accelerator unit 220 retrieves stored instructions from the operation processing queue and executes the queued instructions. In at least one embodiment, these instructions cause the processor to perform a matrix multiplication-accumulation operation on operands loaded into memory at step 602. In at least one embodiment, these instructions are executed in the sequence stored in the operation processing queue.

[0087] In at least one embodiment, in step 608, after the processor completes the operation indicated by the instructions in the operation processing queue, the processing unit identifies whether the loaded operand will be reused in a subsequent operation. In at least one embodiment, the operand may be reused in a series of future steps or operations. In at least one embodiment, whether a given operand will be reused is determined by user-coded software programming that explicitly identifies such operands that will be reused. In at least one embodiment, whether a given operand will be reused is determined by the execution of the operation, such as when a large matrix multiplication operation will be performed using repetitive data as a series of smaller multiplication operations.

[0088] In at least one embodiment, in step 610, the processor performs a barrier wait operation to wait until the current processing phase completes. In at least one embodiment, a phase is considered complete when the processor completes a predetermined number of instructions (such as the number of instructions identified at step 606). In at least one embodiment, once all predetermined instructions have completed, the processor performs a barrier reach operation and indicates or notifies the waiting thread that the memory barrier has been reached. In at least one embodiment, this indication notifies other parts of the processor that the operations have been completed and allows access to, reading, and / or modification of any operand matrices and result matrices of these operations.

[0089] In at least one embodiment, in step 612, if the operand will no longer be reused, the processor loads a new operand into memory in a manner similar to step 602; otherwise, the operand remains unmodified for reuse. In at least one embodiment, process 600 returns to step 604, where the processor uses threads to submit additional instructions to a queue to execute a given operation, and process 600 is repeated.

[0090] In at least one embodiment, process 600 may be generated by a computing device (such as system 100, processor 200, or any suitable system, such as a reference daemon). Figure 8-26C The described computing device is used for execution. In at least one embodiment, through execution process 600, the processor can at least partially execute the reference. Figure 3-5 The described process, or a reference that can be executed. Figure 7 The described application programming interface (API) function, or execution Figure 8-26C Other processes. In at least one embodiment, by performing Figure 6 In process 600, the processor may execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous. In at least one embodiment, by executing Figure 6 In process 600, the processor may use one or more storage queues to store one or more instructions to be executed by one or more portions of one or more streaming multiprocessors (SMs) until said one or more portions become available to execute said one or more instructions. In at least one embodiment, by executing Figure 6 In process 600, the processor can add one or more instructions to one or more memory queues via a single thread in a cooperative thread array (CTA) of one or more SMs. In at least one embodiment, by executing Figure 6 In process 600, the processor can indicate that one or more matrix multiplication-accumulation (MMA) operations have been completed. In at least one embodiment, by executing Figure 6 In process 600, at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator can execute instructions submitted by a thread in the order indicated by one or more storage queues.

[0091] Figure 7 This is a block diagram illustrating a driver and / or runtime according to at least one embodiment, the driver and / or runtime including one or more libraries for providing one or more application programming interfaces (APIs). In at least one embodiment, software program 702 is a software module stored on a processor, such as... Figure 2The processor described herein. In at least one embodiment, the software program 702 includes one or more software modules. In at least one embodiment, the software modules are as described in... Figure 2 Further non-exclusively described herein. In at least one embodiment, one or more APIs 710 are software instruction sets that, if executed, cause one or more processors to perform one or more computational operations. In at least one embodiment, one or more APIs 710 are distributed or otherwise provided as part of one or more libraries 706, runtimes 704, drivers, and / or any other grouping of software and / or executable code further described herein. In at least one embodiment, one or more APIs 710 perform one or more computational operations in response to a call to software program 702. In at least one embodiment, software program 702 is a collection of software code, commands, instructions, or other text sequences for instructing a computing device to perform one or more computational operations and / or calling one or more other instruction sets (such as APIs 710 or API functions 712) to be executed. In at least one embodiment, the functionality provided by one or more APIs 710 includes software functions 712, such as functions that can be used to accelerate one or more portions of software program 702 using one or more parallel processing units (PPUs) (such as graphics processing units (GPUs)). In at least one embodiment, the software program is a compiler.

[0092] In at least one embodiment, API 710 is a hardware interface of one or more circuits for performing one or more computational operations. In at least one embodiment, the one or more software APIs 710 described herein are implemented for performing combined... Figure 1-6 Or one or more circuits using one or more techniques described in 8-26C. In at least one embodiment, one or more software programs 702 include instructions that, if executed, cause one or more hardware devices and / or circuits to perform a combination Figure 1-6 Or one or more of the techniques described above in 8-26C.

[0093] In at least one embodiment, software program 702 (such as a user-implemented software program) utilizes one or more application programming interfaces (APIs) 710 to perform various computational operations, such as memory reservation, matrix multiplication, arithmetic operations, or any computational operation performed by a parallel processing unit (PPU) (such as a graphics processing unit (GPU)), as further described herein. In at least one embodiment, one or more APIs 710 provide a set of callable functions 712 (referred to herein as APIs, API functions, and / or functions), which each perform one or more computational operations, such as computational operations related to parallel computing. In at least one embodiment, one or more APIs 710 provide functions 712 for concurrently executing the tensor instructions described in 716 with one or more other instructions, based at least in part on one or more indicators that at least one tensor instruction is asynchronous.

[0094] In at least one embodiment, one or more software programs 702 interact with or otherwise communicate with one or more APIs 710 to perform one or more computational operations using one or more PPUs (such as GPUs). In at least one embodiment, the one or more computational operations using one or more PPUs include at least one or more sets of computational operations that are accelerated by being performed at least partially by said one or more PPUs. In at least one embodiment, one or more software programs 702 interact with one or more APIs 710 to facilitate parallel computing using remote or local interfaces.

[0095] In at least one embodiment, the interface is software instructions that, when executed, provide access to one or more functions 712 provided by one or more APIs 710. In at least one embodiment, when a software developer compiles one or more software programs 702 in conjunction with one or more libraries 706, the software programs 702 use native interfaces, which include one or more APIs 710 or otherwise provide access to one or more APIs 710. In at least one embodiment, one or more software programs 702 are statically compiled in conjunction with precompiled libraries 706 or uncompiled source code including instructions for executing one or more APIs 710. In at least one embodiment, one or more software programs 702 are dynamically compiled, and the one or more software programs are linked using a linker to one or more precompiled libraries 706 that include one or more APIs 710.

[0096] In at least one embodiment, when a software developer executes a software program that utilizes library 706 (which includes one or more APIs 710) or otherwise communicates with library 706 via a network or other remote communication medium, software program 702 uses a remote interface. In at least one embodiment, one or more libraries 706 including one or more APIs 710 will be executed by a remote computing service (such as a computing resource service provider). In another embodiment, one or more libraries 706 including one or more APIs 710 will be executed by any other computing host that provides said one or more APIs 710 to one or more software programs 702.

[0097] In at least one embodiment, one or more software programs 702 utilize one or more APIs 710 to allocate and otherwise manage memory to be used by the software program 702. In at least one embodiment, one or more software programs 702 utilize one or more APIs 710 to allocate and otherwise manage memory to be used by one or more portions of the software program 702, which will be accelerated using one or more PPUs (such as GPUs or any other accelerators or processors further described herein). These software programs 702 select to deactivate one or more portions of one or more neural networks during training, based at least in part on whether the portions of one or more neural networks will still be used after training the one or more neural networks.

[0098] In at least one embodiment, API 710 is an API for facilitating parallel computing. In at least one embodiment, API 710 is any other API further described herein. In at least one embodiment, API 710 is provided by a driver and / or runtime 704. In at least one embodiment, API 710 is provided by a CUDA user-mode driver. In at least one embodiment, API 710 is provided by a CUDA runtime. In at least one embodiment, the driver is data values ​​and software instructions that, if executed, perform or otherwise facilitate the operation of one or more functions 712 of API 710 during the loading and execution of one or more portions of software program 702. In at least one embodiment, runtime 704 is data values ​​and software instructions that, if executed, perform or otherwise facilitate the operation of one or more functions 712 of API 710 during the execution of software program 702. In at least one embodiment, one or more software programs 702 utilize one or more APIs 710 implemented or otherwise provided by a driver and / or runtime 704 to perform combined arithmetic operations by the one or more software programs 702 during execution by one or more PPUs (e.g., GPUs).

[0099] In at least one embodiment, one or more software programs 702 utilize one or more APIs 710 provided by a driver and / or runtime 704 to perform combinatorial arithmetic operations on one or more PPUs (e.g., GPUs). In at least one embodiment, one or more APIs 710 provide combinatorial arithmetic operations via a driver and / or runtime 704, as described above. In at least one embodiment, one or more software programs 702 utilize one or more APIs 710 provided by a driver and / or runtime 704 to allocate or otherwise reserve one or more blocks of memory 714 for one or more PPUs (e.g., GPUs). In at least one embodiment, one or more software programs 702 utilize one or more APIs 710 provided by a driver and / or runtime 704 to allocate or otherwise reserve blocks of memory. In at least one embodiment, one or more APIs 710 will perform combinatorial arithmetic operations, as incorporated herein by reference. Figure 1-6 As stated above.

[0100] To improve the usability of software program 702 and / or optimize one or more portions of software program 702 for acceleration by one or more PPUs (such as GPUs), in one embodiment, one or more APIs 710 provide one or more API functions 712 to execute, at least in part, the tensor instruction 716 concurrently with one or more other instructions, as described above, and in combination with one or more indicators that at least one tensor instruction is asynchronous. Figure 1-6 Further description. In at least one embodiment, exemplary block diagram 700 depicts a processor including one or more circuits for executing one or more software programs to combine two or more application programming interfaces (APIs) into a single API. In at least one embodiment, exemplary block diagram 700 depicts a system including one or more processors for executing one or more software programs to combine two or more application programming interfaces (APIs) into a single API. In at least one embodiment, the API is used to identify one or more expected software outputs for comparison with one or more other software outputs to be generated by the software.

[0101] In at least one embodiment, Figure 7 The logic and / or processes can be integrated into Figure 1-6 or Figure 8-26C In the systems, processors, and architectures disclosed herein. For example, from... Figure 1-2 The logic / hardware structure can execute at least some or all of the procedures or APIs 300, 400, 500 and / or 600. In at least one embodiment, execution Figure 7 The API disclosed in the document enables one or more circuits to execute the tensor instruction concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous.

[0102] Numerous specific details are set forth in this specification to provide a more thorough understanding of at least one embodiment. However, it will be apparent to those skilled in the art that these inventive concepts can be practiced without one or more of these specific details.

[0103] Data Center

[0104] Figure 8An example data center 800 according to at least one embodiment is illustrated. The data center 800 may include one or more rooms having racks 802 and auxiliary equipment for housing one or more racks 802 and one or more substrates 804. Racks 802 may include one or more substrates 804. Racks 802 may include housings for housing and supporting individual substrates 804. Operational aspects of racks 802 may be adjustable at the rack level (corresponding to a group of substrates 804) or at the substrate level (corresponding to an individual substrate 804), among other options. Racks 802 or substrates 804 may have specific selected maximum operating parameters, such as, but not limited to, power consumption, operating frequency, etc. The data center 800 may be supported by various cooling systems, such as, but not limited to, cooling towers, cooling loops, pumps, and other support systems. The cooling system may include sensors and controllers for monitoring and managing the cooling characteristics of racks 802. The substrates 804 within racks 802 may draw operating power from one or more power distribution units (PDUs; not shown). PDUs can be arranged within racks 802, for example, between racks 802 that include substrates 804, or within racks 802 that also house substrates 804.

[0105] Rack 802 and substrate 804 may include subsystems, modules, add-in cards, and other semiconductor components. Substrate 804 may include one or more computing units 806, each computing unit 806 may include one or more processors 808, one or more memories 810, and an interface controller 812. Computing unit 806 may include any number of processors, such as, but not limited to, a central processing unit (“CPU”), a graphics processing unit (“GPU”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), including any processors described herein, such as, but not limited to, those described herein. Figure 9-21B The processor in the computing unit 806 may include one or more memory storage devices 810 (e.g., dynamic read-only memory, solid-state storage devices, or disk drives), as well as network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power supply modules, and cooling modules, etc. One or more computing units 806 may be a server having one or more of the aforementioned computing resources.

[0106] Computing unit 806 may include individual computing unit groups housed in one or more racks (not shown), or in numerous racks within data centers in different geographical locations (also not shown). Individual computing unit groups may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. Several computing units (e.g., including CPUs and / or other processors) may be grouped within one or more racks to provide computing resources to support one or more workloads. Resource coordinator 814 may configure or otherwise control one or more computing units 806 or groups of computing units. Resource coordinator 814 may include a Software Design Infrastructure (“SDI”) management entity for data center 800. Resource coordinator 814 may include hardware, software, or some combination thereof.

[0107] Data center 800 may include any one or any combination of the framework layer 820, software layer 830, and application layer 840. For example... Figure 8 As shown, framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. Framework layer 820 may include a framework for supporting software 832 of software layer 830 and / or one or more applications 842 of application layer 840. Software 832 or application 842 may respectively include web-based service software or applications, such as, but not limited to, software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 820 may be a type of free and open-source software web application framework, such as, but not limited to, Apache Spark. TM (Hereinafter referred to as "Spark"), which can utilize the distributed file system 828 for large-scale data processing (e.g., "big data"). The job scheduler 822 may include Spark drivers, which facilitate the scheduling of workloads supported by various layers of the data center 800. The configuration manager 824 may be able to configure different layers, such as, but not limited to, the software layer 830 and the framework layer 820 (which includes Spark and the distributed file system 828 for supporting large-scale data processing). The resource manager 826 may be able to manage clustered or grouped compute units 806 mapped to or allocated to support the distributed file system 828 and the job scheduler 822. The resource manager 826 may coordinate with the resource coordinator 814 to manage these mapped or allocated compute resources.

[0108] Software 832 may be included in software layer 830, and may include software used by at least a portion of computing units 806, one or more computing units 806, groups of computing units 806, and / or the distributed file system 828 of framework layer 820. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0109] Application 842 may be included in application layer 840 and may include one or more types of applications used by at least the portions of computing unit 806, one or more computing units 806, groups of computing units 806, and / or the distributed file system 828 of framework layer 820. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing applications, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0110] Any of the Configuration Manager 824, Resource Manager 826, and Resource Coordinator 814 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can alleviate the burden on data center operators of Data Center 800 to make potentially erroneous configuration decisions and may prevent underutilized and / or poorly performing portions of the data center.

[0111] Data center 800 may include tools, services, software, or other resources for training one or more machine learning models according to one or more embodiments described herein, or for using one or more machine learning models to predict or infer information. For example, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above regarding data center 800. The trained machine learning model corresponding to one or more neural networks can be used with the resources described above regarding data center 800 to infer or predict information using weight parameters calculated through one or more training techniques described herein.

[0112] Data Center 800 can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware (e.g., Figure 9-21BThe embodiments described herein can be used to perform some or all of the processes and techniques described elsewhere, such as, but not limited to, training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow a user to train or perform information inference, such as, but not limited to, image recognition, speech recognition, or other artificial intelligence services.

[0113] In at least one embodiment, processor 808 may include one or more processors and / or one or more circuits for: executing the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or executing one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled in program order before one or more pause instructions; adding one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicating to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or executing one or more instructions in an order independent of the order in which the instructions were submitted by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator, or otherwise performing any of the operations described above or elsewhere herein. In at least one embodiment, processor 808 is configured by software 832 to: execute tensor instructions concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are programmatically preceding one or more pause instructions; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted by at least one of the processing cores or MMA accelerators of a graphics processing unit (GPU), or otherwise perform any of the operations described above or elsewhere herein. Data center 800 may use logic, CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware (e.g., Figure 9-21B (The embodiments in the document) can be used to perform any of the operations described above or elsewhere in this document.

[0114] processor

[0115] The following figures illustrate, but are not limited to, example processors and processing systems that can be used to: execute tensor instructions concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform some or all of the processes, operations and / or techniques described elsewhere herein. Example processors and processing systems may be software-configurable to: execute tensor instructions concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are programmatically preceding one or more pause instructions; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. Processors and processing systems may include logic, central processing units (CPUs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), XPUs (i.e., any computing architecture best suited to the needs of the application), or other hardware (e.g., Figure 9-21BThe embodiments described herein are used to perform any of the operations described above, below, or elsewhere herein. The processor and / or processing system described herein may include one or more circuits that can be used to: execute a tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are programmatically preceding one or more pause instructions; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or perform any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted. As used herein, one or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted. Figure 26A and Figure 26B The illustration depicts logic 2615 according to at least one embodiment, which, as described elsewhere herein, can be used in one or more devices to perform operations such as, but not limited to, those discussed herein. For example, logic can refer to any combination of software logic, hardware logic, and / or firmware logic that provides the functionality and / or operations described herein, wherein the logic can be collectively or individually embodied as part of a circuit system forming a larger system, such as an integrated circuit (IC), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), or one or more processors (e.g., CPU, GPU).

[0116] Figure 9 A processor according to at least one embodiment is illustrated, which is a system-on-a-chip (SOC) 900 (may be referred to as a system-on-a-chip, superchip, or other names). The SOC 900 may include processor complexes 910 and 940. The SOC 900 may include any number of processor complexes 910 and / or processor complexes 940, which may include any number of processors described herein in any combination, such as, but not limited to, processors described herein in any combination. Figure 9-21B The processor in the SOC 900 may include, for example, a central processing unit (CPU) and a graphics processor (GPU). Alternatively, processor 910 may include a GPU, and processor 940 may include a GPU. The SOC 900 may include any number of display controllers 992, any number of multimedia engines 994, any number of I / O interfaces 970, any number of memory controllers 980, and any number of fabrics 960 in any combination. For ease of explanation, this document uses reference numbers to identify objects and bracket numbers to identify instances (if necessary) to denote multiple instances of similar objects. The SOC 900 may include a processor from Broadcom Inc., Palo Alto, California.

[0117] Processor complex 910 may include a CPU, processor complex 940 may include a GPU, and SOC 900 may include a processing unit integrating processor complex 910 and processor complex 940 onto a single chip. Certain tasks may be assigned to processor complex 910, while other tasks may be assigned to processor complex 940. Processor complex 910 may be configured to execute main control software associated with SOC 900, such as, but not limited to, an operating system. Processor complex 910 may be the main processor of SOC 900, controlling and coordinating the operation of other processors. Processor complex 910 may issue commands that control the operation of processor complex 940 to perform some or all of the operations described herein. Processor complex 910 may be configured to execute host-executable code derived from CUDA or other source code (e.g., HIP source code), while processor complex 940 may be configured to execute device-executable code derived from CUDA or other source code to perform any of the operations described herein.

[0118] The processor complex 910 may include cores 920(1)-920(4) and cache (e.g., L3 cache) 930 for storing information for performing the operations described herein. The processor complex 910 may include any number of cores 920 in any combination and any number and type of cache. The cores 920 may be configured to execute instructions of a specific instruction set architecture (“ISA”) to perform some or all of the operations described herein. Each core 920 may include a CPU core. Cores 920(1)-920(4) may be referred to as compute units or arithmetic units. The SOC 900 may include any number of processor complexes 910, architecture 960, I / O interface 970, and memory controller 980.

[0119] Each core 920 may include a fetch / decode unit 922, an integer execution engine 924, a floating-point execution engine 926, and an L2 cache 928. The fetch / decode unit 922 may fetch instructions to perform some or all of the operations described herein (e.g., but not limited to APIs compiled into instructions) and decode those instructions, generate micro-operations, and dispatch individual micro-instructions to the integer execution engine 924 and / or the floating-point execution engine 926. The fetch / decode unit 922 may concurrently dispatch one micro-instruction to the integer execution engine 924 and another micro-instruction to the floating-point execution engine 926. The integer execution engine 924 may perform integer and memory operations. The floating-point engine 926 may perform floating-point and vector operations. The fetch / decode unit 922 may dispatch micro-instructions to one or more execution engines, which may replace both the integer execution engine 924 and the floating-point execution engine 926.

[0120] Each core 920(i) (where i is an integer representing a specific instance of core 920) can access the L2 cache 928(i) included in core 920(i). Each core 920 included in core complex 910(j) (where j is an integer representing a specific instance of core complex 910) can be connected to other cores 920 included in core complex 910(j) via the L3 cache 930(j) included in core complex 910(j). The cores 920 included in core complex 910(j) (where j is an integer representing a specific instance of core complex 910) can access all L3 caches 930(j) included in core complex 910(j). The L3 cache 930 can include any number of slices.

[0121] Processor complex 940 may be a graphics complex that can be configured to perform computational operations (e.g., the computational operations described herein) in a highly parallel manner. Processor complex 940 may be configured to perform graphics pipeline operations, such as, but not limited to, drawing commands, pixel operations, geometric calculations, and other operations associated with rendering an image to a display. Processor complex 940 may be configured to perform graphics-independent operations, such as, but not limited to, neural network training and / or simulation. Processor complex 940 may be configured to perform both graphics-related and graphics-independent operations.

[0122] The processor complex 940 may include any number of compute units 950(1)-950(N) (where N is any integer greater than 1) and an L2 cache 942. The compute units 950 may share the L2 cache 942, which may store information that will be used to perform some or all of the operations described herein. The L2 cache 942 may be partitioned. The processor complex 940 may include any number of compute units 950 and any number (including zero) and type of cache. The processor complex 940 may include any number of dedicated graphics hardware.

[0123] Each compute unit 950 may include any number of SIMD units 952(1)-952(N) (where N is any integer greater than 1) and shared memory 954. Each SIMD unit 952 may implement a SIMD architecture and may be configured to perform some or all of the operations described herein in parallel. Each compute unit 950 may execute any number of thread blocks, but each thread block may execute on a single compute unit 950, although in some embodiments, the thread block may execute on multiple compute units. A thread block may include any number of execution threads. A workgroup may be a thread block. Each SIMD unit 952 may execute a set of threads. A set of threads (e.g., 16 threads), also referred to as a warp, subgroup, or wavefront (e.g., used by AMD and Intel), may belong to a single thread block and be configured to process different datasets based on a single instruction set. Prediction may be used to disable one or more threads in a warp, subgroup, or wavefront. A lane may be a thread. A work item can be a thread, such as (but not limited to) an OpenCL thread. Different thread bundles, subgroups, or wavefronts within a thread block can be synchronized together and communicate via shared memory 954. Each compute unit 950 can include one or more thread block clusters, where thread block clusters can implement programmable control over locality at a larger granularity than a single thread block of a single streaming multiprocessor (SM). Thread block clusters (also referred to as “clusters”) can support multiple thread blocks running concurrently across streaming multiprocessors, thereby synchronously and cooperatively acquiring, exchanging, or otherwise using data. In at least one embodiment, a streaming multiprocessor (“SM”) can refer to a streaming microprocessor, a streaming processor (“SP”), a streaming processing unit (“SPU”), a compute unit (“CU”), an execution unit (“EU”), and / or a slice, where a slice in this context can refer to a portion of the processing resources within a processing unit (e.g., 16 cores, a ray tracing unit, a thread bootstrap, or a scheduler).

[0124] Structure 960 may be a system interconnect that facilitates data and control transfers across processor complex 910, processor complex 940, I / O interface 970, memory controller 980, display controller 992, and multimedia engine 994, for example, to perform some or all of the operations described herein. SOC 900 may include any number and type of system interconnects other than or replacing structure 960, facilitating data and control transfers across any number and type of directly or indirectly linked components within or outside SOC 900. I / O interface 970 may represent any number and type of I / O interfaces (e.g., PCI, PCI extensions (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices may be coupled to I / O interface 970. Peripherals that may be coupled to I / O interface 970 may include keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, etc.

[0125] Display controller 992 can display images on one or more display devices, such as, but not limited to, liquid crystal displays (“LCD”) devices. Multimedia engine 994 can include any number and type of circuitry related to multimedia, such as, but not limited to, video decoders, video encoders, image signal processors, etc. Memory controller 980 can facilitate data transfer between SOC 900 and unified system memory 990. Processor complex 910 and processor complex 940 can share unified system memory 990. Unified system memory 990 can include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as, but not limited to, synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Unified system memory 990 can include 3D stacked memory, including but not limited to high bandwidth memory (HBM), HBM2e, or HDM3.

[0126] The SOC 900 can implement a memory subsystem comprising any number and type of memory controllers 980 and memory devices (e.g., shared memory 954), which may be dedicated to a single component or shared among multiple components to perform any of the operations described herein. The SOC 900 can implement a cache subsystem comprising one or more cache memories (e.g., L2 cache 928, L3 cache 930, and L2 cache 942), each cache memory which may be dedicated to any number of components (e.g., core 920, core complex 910, SIMD unit 952, compute unit 950, and processor complex 940) or shared among any number of components (e.g., core 920, core complex 910, SIMD unit 952, compute unit 950, and processor complex 940).

[0127] In at least one embodiment, the SOC 900 may include one or more circuitry configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuitry, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0128] Figure 10A A parallel processor 1000 according to at least one embodiment is illustrated. The parallel processor 1000 can be implemented using one or more circuits and can be referred to as a programmable processor (e.g., CPU and / or GPU), logic, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other hardware (e.g., Figure 9-21B (The embodiments in the document) are used to perform any of the operations described above or elsewhere herein.

[0129] Parallel processor 1000 may include parallel processing unit 1002 for performing any of the operations described above or elsewhere herein. Parallel processing unit 1002 may include I / O unit 1004 that enables communication with other devices, including other instances of parallel processing unit 1002. I / O unit 1004 may be directly connected to other devices. I / O unit 1004 may be connected to other devices via the use of a hub or switch interface, such as, but not limited to, memory hub 1005. The connection between memory hub 1005 and I / O unit 1004 may form a communication link 1013. I / O unit 1004 may be connected to host interface 1006 and memory crossbar switch 1016, wherein host interface 1006 receives commands directed to perform processing operations, and memory crossbar switch 1016 receives commands directed to perform memory operations.

[0130] When host interface 1006 receives a command buffer via I / O unit 1004, host interface 1006 can route the work operations that execute these commands to front-end 1008. Front-end 1008 can be coupled to scheduler 1010 (which may be referred to as sequencer), which is configured to distribute commands or other work items to processing cluster array 1012. Scheduler 1010 can ensure that processing cluster array 1012 is correctly configured and in an active state before tasks are distributed to the cluster of processing cluster array 1012. Scheduler 1010 can be implemented via firmware logic executed on a microcontroller. The microcontroller-implemented scheduler 1010 can be configured to perform complex scheduling and work distribution operations at both coarse and fine granular levels, thereby enabling fast preemption and context switching of threads executing on processing array 1012. Host software can validate workloads scheduled on processing cluster array 1012 via one of multiple graphics processing paths. The workload can then be automatically distributed to the processing array cluster 1012 by the scheduler 1010 logic within the microcontroller, which includes the scheduler 1010.

[0131] Processing cluster array 1012 can perform any of the operations described above or elsewhere herein, and may include up to “N” processing clusters (e.g., clusters 1014A, 1014B through 1014N), where “N” represents a positive integer (which may be a different integer “N” than used in other diagrams). Each cluster 1014A-1014N in processing cluster array 1012 can execute a large number of concurrent threads. Scheduler 1010 may use various scheduling and / or work distribution algorithms to distribute work to clusters 1014A-1014N in processing cluster array 1012, which may vary depending on the workload generated by each type of program or computation. Scheduling may be dynamically handled by scheduler 1010 or may be assisted by compiler logic during the compilation of program logic configured to be executed by processing cluster array 1012. Different clusters 1014A-1014N of processing cluster array 1012 may be assigned to process different types of programs or perform different types of computations.

[0132] The processing cluster array 1012 can be configured to perform various types of parallel processing operations, such as, but not limited to, any of the operations described above or elsewhere herein. The processing cluster array 1012 can be configured to perform general-purpose parallel computing operations. For example, the processing cluster array 1012 may include logic for performing processing tasks, including filtering video and / or audio data, performing modeling operations (including physical operations), and performing data transformations.

[0133] Processing cluster array 1012 can be configured to perform parallel graphics processing operations. Processing cluster array 1012 may include additional logic for supporting the execution of such graphics processing operations, including but not limited to texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. Processing cluster array 1012 can be configured to execute shader programs related to graphics processing, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. Parallel processing unit 1002 can transfer data from system memory via I / O unit 1004 for processing. During processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 1022) during processing and then written back to system memory.

[0134] When the parallel processing unit 1002 is used to perform graphics processing, the scheduler 1010 can be configured to divide the processing workload into tasks of approximately equal size to better distribute graphics processing operations to multiple clusters 1014A-1014N of the processing cluster array 1012. Each part of the processing cluster array 1012 can be configured to perform different types of processing. For example, a first part can be configured to perform vertex shading and topology generation, a second part can be configured to perform tessellation and geometry shading, and a third part can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. Intermediate data generated by one or more clusters 1014A-1014N can be stored in a buffer to allow intermediate data to be transferred between clusters 1014A-1014N for further processing.

[0135] Processing cluster array 1012 can receive processing tasks to be executed via scheduler 1010, which receives commands defining the processing tasks from front end 1008. Processing tasks may include indexes of data to be processed, such as surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands defining how to process the data (e.g., which program to execute). Scheduler 1010 can be configured to retrieve the index corresponding to the task, or can receive the index from front end 1008. Front end 1008 can be configured to ensure that processing cluster array 1012 is configured to be active before the workload specified by the incoming command buffer (e.g., batch buffer, push buffer, etc.) is initiated.

[0136] Each instance of one or more instances of parallel processing unit 1002 may be coupled to parallel processor memory 1022 to perform any of the operations described above or elsewhere herein. Parallel processor memory 1022 may be accessed via memory crossbar switch 1016, which may receive memory requests from processing cluster array 1012 and I / O unit 1004. Memory crossbar switch 1016 may access parallel processor memory 1022 via memory interface 1018. Memory interface 1018 may include multiple partition units (e.g., partition units 1020A, 1020B through 1020N), each partition unit may be coupled to a portion (e.g., a memory cell) of parallel processor memory 1022. The number of partition units 1020A-1020N can be configured to be equal to the number of memory units, such that the first partition unit 1020A has a corresponding first memory unit 1024A, the second partition unit 1020B has a corresponding memory unit 1024B, and the Nth partition unit 1020N has a corresponding Nth memory unit 1024N. The number of partition units 1020A-1020N may not be equal to the number of memory units.

[0137] Memory cells 1024A-1024N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as, but not limited to, synchronous graphics random access memory (SGRAM), which includes graphics double data rate (GDDR) memory. Memory cells 1024A-1024N may also include 3D stacked memory, including but not limited to high-bandwidth memory (HBM), HBM2e, or HDM3. Render targets (e.g., but not limited to framebuffers or texture maps) may be stored in memory cells 1024A-1024N, allowing partitioning cells 1020A-1020N to write portions of each render target in parallel to efficiently utilize the available bandwidth of the parallel processor memory 1022. A local instance of the parallel processor memory 1022 may not be included to support a unified memory design that combines system memory with local cache memory.

[0138] Any cluster 1014A-1014N in the processing cluster array 1012 can process data to be written to any memory cell 1024A-1024N within the parallel processor memory 1022. The memory crossbar switch 1016 can be configured to transfer the output of each cluster 1014A-1014N to any partition cell 1020A-1020N, ​​or to another cluster 1014A-1014N on which additional processing operations can be performed. Each cluster 1014A-1014N can communicate with the memory interface 1018 via the memory crossbar switch 1016 to read from or write to various external memory devices. The memory crossbar switch 1016 can be connected to the memory interface 1018 to communicate with the I / O unit 1004, or to a local instance of the parallel processor memory 1022, enabling processing units within different processing clusters 1014A-1014N to communicate with system memory or other memory local to the non-parallel processing unit 1002. The memory crossbar switch 1016 can use virtual channels to separate traffic flows between clusters 1014A-1014N and partition units 1020A-1020N.

[0139] Multiple instances of the parallel processing unit 1002 can be mounted on a single add-in card, or multiple add-in cards can be interconnected. Even if different instances of the parallel processing unit 1002 have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences, these different instances can be configured to interoperate. For example, some instances of the parallel processing unit 1002 may include higher-precision floating-point units relative to other instances. Systems including one or more instances of the parallel processing unit 1002 or the parallel processor 1000 can be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0140] Figure 10A It also includes a block diagram of a partitioning unit 1020 according to at least one embodiment. The partitioning unit 1020 is... Figure 10AAn example of one of the partition units 1020A-1020N in the parallel processor memory. Partition unit 1020 may include L2 cache 1021, frame buffer interface 1025, and ROP 1026 (raster operation unit). L2 cache 1021 may be a read / write cache configured to perform load and store operations received from memory crossbar switch 1016 and ROP 1026. Read misses and urgent write-back requests may be output from L2 cache 1021 to frame buffer interface 1025 for processing. Updates may also be sent to the frame buffer via frame buffer interface 1025 for processing. Frame buffer interface 1025 may interface with one of the memory cells in the parallel processor memory, such as, but not limited to, Figure 10A The memory cells 1024A-1024N (shown as 1024) are located in the parallel processor memory 1022 (for example).

[0141] ROP 1026 can be a processing unit that performs raster operations, such as, but not limited to, stenciling, z-testing, blending, etc. ROP 1026 can then output processed graphics data stored in graphics memory. ROP 1026 may include compression logic for compressing depth or color data written to memory and decompressing depth or color data read from memory. The compression logic can be lossless compression logic that utilizes one or more compression algorithms. The type of compression performed by ROP 1026 can vary based on the statistical characteristics of the data to be compressed. For example, incremental color compression is performed on depth and color data on a per-tile basis.

[0142] ROP 1026 can be included in each processing cluster (e.g., Figure 10A The data is stored within clusters 1014A-1014N, not within partition units 1020. Read and write requests for pixel data (not pixel fragment data) can be transferred via memory crossbar switch 1016. Processed graphics data can be displayed on a monitor and routed for further processing by the processor, or routed to... Figure 10A One of the processing entities within the 1000 parallel processors in the system is further processed.

[0143] In at least one embodiment, the parallel processor 1000 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0144] Figure 10B A block diagram including a processing cluster 1014 within a parallel processing unit according to at least one embodiment. The processing cluster may be... Figure 10A An instance of one of the processing clusters 1014A-1014N is provided, which can be used to perform any of the operations described above or elsewhere herein. Processing cluster 1014 can be configured to execute many threads in parallel, where a “thread” refers to an instance of a specific program executed on a specific input dataset. Single Instruction Multiple Data (SIMD) instruction issuing techniques can be used to support the parallel execution of a large number of threads without providing multiple independent instruction units. Single Instruction Multiple Thread (SIMT) techniques can be used to support the parallel execution of a large number of typically synchronous threads using a common instruction unit configured to issue instructions to a set of processing engines within each processing cluster.

[0145] The operation of cluster 1014 can be controlled via pipeline manager 1032, which distributes processing tasks to SIMT parallel processors. Pipeline manager 1032 can... Figure 10A The scheduler 1010 receives instructions and manages the execution of these instructions via the graphics multiprocessor 1034 and / or texture unit 1036. The graphics multiprocessor 1034 may be an example instance of a SIMT parallel processor. However, the processing cluster 1014 may include various types of SIMT parallel processors with different architectures. The processing cluster 1014 may include one or more instances of the graphics multiprocessor 1034. The graphics multiprocessor 1034 can process data and can use the data cross switch 1040 to distribute the processed data to one of several possible destinations, including other shader units. The pipeline manager 1032 can facilitate the distribution of processed data by specifying the destination of the processed data to be distributed via the data cross switch 1040.

[0146] Each graphics multiprocessor 1034 within the processing cluster 1014 may include a set of identical functional execution logic (e.g., arithmetic logic units, load-memory units, etc.) for performing computations for any of the operations described above or elsewhere herein. The functional execution logic can be configured in a pipelined manner, where new instructions can be issued before previous instructions complete. The functional execution logic can support a wide range of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and computation of various algebraic functions. Different operations can be performed using the same functional unit hardware, and arbitrary combinations of functional units are possible.

[0147] Instructions transmitted to the processing cluster 1014 can form threads, which may also be called thread bundles, subgroups, waves, or wavefronts. A group of threads executing across a set of parallel processing engines can be called a thread group. Thread groups can execute a common program on different input data. Each thread within a thread group can be assigned to a different processing engine within the graphics multiprocessor 1034. The number of threads in a thread group can be less than the number of processing engines within the graphics multiprocessor 1034. When the number of threads in a thread group is less than the number of processing engines, one or more processing engines may be idle during the processing cycle of that thread group. The number of threads in a thread group can also be more than the number of processing engines within the graphics multiprocessor 1034. When the number of threads in a thread group is more than the number of processing engines within the graphics multiprocessor 1034, processing can be performed in consecutive clock cycles. Multiple thread groups can execute concurrently on the graphics multiprocessor 1034.

[0148] The graphics multiprocessor 1034 includes an internal cache memory for performing load and store operations, such as, but not limited to, any of the operations described above or elsewhere herein. The graphics multiprocessor 1034 may forgo the internal cache and instead use a cache memory within the processing cluster 1014 (e.g., L1 cache 1048). Each graphics multiprocessor 1034 may also access partition units that can be shared across all processing clusters 1014 (e.g., ...). Figure 10A The L2 cache within partition units 1020A-1020N is used for transferring data between threads. The graphics multiprocessor 1034 can also access off-chip global memory, which may include one or more of the local parallel processor memory and / or system memory. Any memory outside the parallel processing unit 1002 can be used as global memory. The processing cluster 1014 may include multiple instances of the graphics multiprocessor 1034 and can share common instructions and data, which can be stored in the L1 cache 1048.

[0149] Each processing cluster 1014 may include an MMU 1045 (Memory Management Unit), which can be configured to map virtual addresses to physical addresses. One or more instances of the MMU 1045 may reside in... Figure 10A The MMU 1045 is located within the memory interface 1018. It may include a set of page table entries (PTEs) for mapping virtual addresses to physical addresses of tiles, and optional cache line indexes. The MMU 1045 may include address translation lookup buffers (TLBs) or caches that may reside within the graphics multiprocessor 1034 or L1 1048 cache or processing cluster 1014. Physical addresses can be processed to distribute surface data access locally, allowing for efficient request interleaving between partition units. The cache line indexes can be used to determine whether a request for a cache line is a hit or a miss.

[0150] Processing cluster 1014 can be configured such that each graphics multiprocessor 1034 is coupled to a texture unit 1036 for performing texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. Texture data can be read from an internal texture L1 cache (not shown) or an L1 cache within the graphics multiprocessor 1034, and can be retrieved as needed from an L2 cache, local parallel processor memory, or system memory. Each graphics multiprocessor 1034 can output processed tasks to a data crossbar switch 1040 to provide the processed tasks to another processing cluster 1014 for further processing, or store the processed tasks in an L2 cache, local parallel processor memory, or system memory via a memory crossbar switch 1016. Pre-ROP 1042 (pre-raster operation unit) can be configured to receive data from the graphics multiprocessor 1034 and direct the data to ROP units, which can be associated with partitioning units (e.g., ...) described herein. Figure 10A The PreROP 1042 unit is located together with partition units 1020A-1020N. The PreROP 1042 unit can perform color blending optimization, organize pixel color data, and perform address translation.

[0151] In at least one embodiment, the processing cluster 1014 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more storage queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0152] Figure 10CA graphics multiprocessor 1034 according to at least one embodiment is illustrated, for example, to perform any of the operations described above or elsewhere herein. The graphics multiprocessor 1034 may be coupled to a pipeline manager 1032 of a processing cluster 1014. The graphics multiprocessor 1034 may include an execution pipeline including, but not limited to, an instruction cache 1052 (e.g., which may store instructions, such as, but not limited to, compiled API instructions), instruction units 1054, address mapping units 1056, register files 1058, one or more general-purpose graphics processing unit (GPGPU) cores 1062, and one or more load / store units 1066, one or more of which may perform load / store operations to load / store instructions corresponding to the execution operations. The GPGPU cores 1062 and load / store units 1066 may be coupled to cache memory 1072 and shared memory 1070 via a memory and cache interconnect 1068. The GPGPU cores 1062 may be part of a SoC, such as, but not limited to, [other components]. Figure 9 It is part of the integrated circuit 900.

[0153] Instruction cache 1052 can receive a stream of instructions to be executed (e.g., perform any of the operations described above or elsewhere herein) from pipeline manager 1032. Instructions can be cached in instruction cache 1052 and dispatched for execution by instruction unit 1054. Instruction unit 1054 can dispatch instructions into thread groups (e.g., thread bundles, subgroups, wavefronts, or waves), with each thread in the thread group assigned to a different execution unit within GPGPU core 1062. Instructions can access any of the local, shared, or global address spaces by specifying an address within a unified address space. Address mapping unit 1056 can be used to translate addresses in the unified address space into different memory addresses accessible by load / store unit 1066.

[0154] Register file 1058 provides a set of registers for the functional units of graphics multiprocessor 1034. Register file 1058 provides temporary storage for operands on data paths connected to functional units of graphics multiprocessor 1034 (e.g., GPGPU core 1062, load / store unit 1066). Register file 1058 can be partitioned among the functional units, such that each functional unit is allocated a dedicated portion of register file 1058. Register file 1058 can be partitioned among different thread bundles (which may be referred to as wavefronts, subgroups, and / or waves or threads) executed by graphics multiprocessor 1034.

[0155] Each GPGPU core 1062 may include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) for executing instructions of the graphics multiprocessor 1034. The architectures of the GPGPU cores 1062 may be similar or different. A first part of the GPGPU core 1062 may include a single-precision FPU and an integer ALU, while a second part of the GPGPU core may include a double-precision FPU. The FPU may implement IEEE 754-2008 standard floating-point arithmetic or enable variable-precision floating-point arithmetic. The graphics multiprocessor 1034 may also include one or more fixed-function or special-function units for performing specific functions, such as, but not limited to, copying rectangles or pixel blending operations. One or more of the GPGPU cores 1062 may also include fixed-function or special-function logic.

[0156] The GPGPU core 1062 may include SIMD logic capable of executing a single instruction on multiple sets of data. The GPGPU core 1062 can physically execute SIMD4, SIMD8, and SIMD16 instructions, and logically execute SIMD1, SIMD2, and SIMD32 instructions. The SIMD instructions for the GPGPU core may be generated by the shader compiler at compile time, or may be automatically generated when executing programs written and compiled for Single Program Multiple Data (SPMD) or SIMT architectures. Multiple threads of a program can be configured for a SIMT execution model that can be executed via a single SIMD instruction. For example, eight SIMT threads performing the same or similar operations can be executed in parallel via a single SIMD8 logic unit.

[0157] The memory and cache interconnect 1068 may include an interconnect network that connects each functional unit of the graphics multiprocessor 1034 to the register file 1058 and shared memory 1070. The memory and cache interconnect 1068 may be a cross-switch interconnect that allows the load / store unit 1066 to perform load and store operations between the shared memory 1070 and the register file 1058. The register file 1058 may operate at the same frequency as the GPGPU core 1062, thus data transfer between the GPGPU core 1062 and the register file 1058 can have very low latency. The shared memory 1070 can be used to implement communication between threads executing on functional units within the graphics multiprocessor 1034. The cache memory 1072 can be used as a data cache, for example, for caching texture data transferred between functional units and texture units 1036. The shared memory 1070 can also be used as a program-managed cache. In addition to automatically caching the data stored in the cache memory 1072, threads executing on the GPGPU core 1062 can also programmatically store data in shared memory.

[0158] The parallel processor or GPGPU described herein can be communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., high-speed interconnects, such as, but not limited to, PCIe or NVLink). A System-on-a-Chip (SoC) may include the parallel processor or GPGPU described herein, which executes on the SoC. The GPU may be integrated as a core on a package or chip and communicatively coupled to the core via an internal processor bus / interconnect within the package or chip. Regardless of the GPU's connection method, the processor core can assign work to the GPU in the form of a sequence of commands / instructions contained in a job descriptor. The GPU can then use dedicated circuitry / logic to efficiently process these commands / instructions to perform any of the operations described above or elsewhere herein.

[0159] In at least one embodiment, the graphics multiprocessor 1034 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0160] Figure 11A processor 1100 according to at least one embodiment is illustrated. The processor 1100 may include a hybrid architecture processor (e.g., Lunar Lake or Meteor Lake) from Intel Corporation, Santa Clara, California, or other processors sharing at least some of the components described herein. The processor 1100 may include one or more central processing units (CPU 1102), one or more graphics processing units (GPU 1106), and / or one or more neural processing units (NPU 1108), which may be, for example, dedicated AI accelerators for offloading artificial intelligence (AI) workloads from the CPU 1102 and GPU 1106. The processor 1100 may use instructions that, if executed, cause the processor 1100 and / or any of its components to perform some or all of the processes and techniques described elsewhere herein. The processor 1100 may include any number of memory and cache units 1110 for facilitating processing between different components of the processor 1100. The memory and cache 1110 on processor 1100 may include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. Regarding processor 1100 and any components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to processor 1100. The results of APIs may be stored in storage devices internal or external to processor 1100, including registers, DRAM, flash memory, SRAM, cache, or other memory. One or more APIs described herein may include calls.

[0161] Processor 1100 may include a computing engine as CPU 1102, and may include any number of cores, such as, but not limited to, up to 16 cores / 22 threads. The cores in CPU 1102 may include P-cores (performance), E-cores (high efficiency), and LP-E cores (low-power, high-efficiency). Performance cores can be used for low-latency, single-threaded, computationally intensive workloads, while high-efficiency cores can be used for multi-threaded, less computationally intensive workloads. Low-power, high-efficiency cores can be used for scalable multi-threaded execution and offloading background tasks. P-cores can be used for single-threaded and limited-threaded execution, while E-cores and LP-E cores are used for multi-threaded throughput and power efficiency.

[0162] The GPU 1106 can include any number of graphics engines, such as, but not limited to, those with 8 Xe cores (up to 128 execution units or EUs). Arc TM Graphics engine (Xe LPG). For example... Figure 11 As shown, GPU 1106 may include a vector engine 1110 and a matrix engine 1112, which, for example, can run FP, INT, and matrix operation tasks simultaneously, individually, or in batches. GPU 1106 may include a load / store unit 1114, as well as other memories, such as, but not limited to, an instruction cache (I$) 1116 and an L1 cache / subsystem local memory (SLM) 1118, which may, for example, store instructions for performing any of the operations described above or elsewhere herein.

[0163] The NPU 1104 may include one or more AI Boost integrates a Neural Processing Unit (NPU). The NPU 1104 can be enumerated as an integrated PCIe device to the host processor. The NPU 1104 may include one or more (e.g., two) Neural Computation Engine (NCE) tiles 1130. Each tile may be configured with any combination of, but not limited to, the following: (e.g., 2000) Multiply-Accumulate (MAC) engines 1134, a post-processing engine (not shown), an AEP processor (not shown), and memory per tile (2MB dedicated SRAM), such as... Figure 11 As shown. For general computing needs, the neural computing engine 1130 may include a disturbance pipeline 1132, an activation function (AF) 1136, a data transformation 1138, a load / store 1140, and a streaming hybrid architecture vector engine (SHAVE) 1128 for high-performance parallel computing, which may include a DMA (Direct Memory Access) engine 1124 for transporting data between system memory DRAM (Dynamic Random Access Memory) 1126 and a software-managed cache. The built-in device MMU (Memory Management Unit) 1122, plus the IOMMU (Input-Output Memory Management Unit) (not shown), can support multiple concurrent hardware contexts and provide secure isolation between execution contexts according to the MCDM (Microsoft Computing Driver Model) architecture. The processor 1100 may also include a media unit (not shown), which may be included on or separate from the XCD or other components of the processor 1100 to enable video playback and video processing of compressed or uncompressed data, such as using HEVC, AV1, VP9, ​​and AVC hardware-accelerated decoding support and HEVC, VP9, ​​and AVC hardware-accelerated encoding support.

[0164] Thread bootstrap ( The Thread Director (which includes firmware built into the processor 1100) can prioritize and manage the distribution of workloads, thereby sending tasks to optimized cores. For example, the thread director can tie P cores, E cores, and / or LP-E cores (as described above) together with task scheduling capabilities and the ability to send less demanding tasks to E cores or LP-E cores. Deep learning acceleration ( DLBoost (not shown) can provide built-in AI acceleration for training and inference workloads and may include support for VNNI (for CPU) and DP4a (for GPU) instruction sets. This instruction set can be used with OpenVINO. TM The toolkit and oneAPI are optimized to accelerate INT8 inference. For example, the software stack described elsewhere in this document can be used to leverage OpenVINO. TM The toolkit enables AI inference. The processor 1100 can be configured to execute applications, such as, but not limited to, CUDA programs.

[0165] In at least one embodiment, processor 1100 may include one or more circuitry configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuitry, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0166] Processor 1100 may alternatively include a processor based on Qualcomm's AIEngine Direct architecture from Santa Clara, California, or other processors sharing at least some of the components described herein. It may include any number of NPUs, GPUs, CPUs, and other associated components, such as, but not limited to, an NPU 1104 as a Hexagon NPU, a GPU 1106 as an Adreno GPU, a CPU 1102 as a Kryo or Qualcomm Oryon CPU, and a Qualcomm Sensing Hub (not shown) and a memory subsystem 1110. Hexagon NPU 1104 may include power rails, micro-tile inference units, hardware acceleration units, tensor units, scalar units, and vector units (all not shown), which may have dedicated or shared memory (e.g., cache or memory, such as HBM3) for storing, for example, instructions for performing any of the operations described above or elsewhere herein. The Adreno GPU 1106 can provide graphics and parallel processing for AI, in formats including but not limited to 32-bit floating-point (FP32), 16-bit floating-point (FP16), and 8-bit integer (INT8). The Kryo or Qualcomm Oryon CPU 1102 can execute AI workloads and handle the contextualization of ubiquitous generative AI applications. The CPU 1102 may also include an instruction fetch unit, a renaming and deprecation unit, a memory management unit, a vector execution unit, an integer execution unit, and a load and store unit for processing and instruction management. Regarding the processor 1100 and any of its components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by the instruction fetch unit, decoded by the processor decoder or equivalent, scheduled (e.g., sequentially or out of order) for execution by the scheduler or equivalent, executed by execution logic or equivalent, reordered, and then deprecated by the renaming and deprecation unit. The API (and / or compiled instructions including the API) can be stored in any storage device (e.g., cache and / or memory) inside or outside the processor 1100. An arbitrary number of CPU cores 1102 can be included in an arbitrary number of CPU clusters, which can be coupled to memory and / or cache, such as, but not limited to, a shared L2 cache. Memory can be separate or shared; for example, the CPU clusters of CPU cores 1102 can be coupled to a memory subsystem 1110, which can include structures capable of reading and writing to memory (e.g., DRAM), system-level caches, and an arbitrary number of memory management units.The Qualcomm sensing hub (not shown) includes a miniature NPU, power rails, and conventional sensors (such as gyroscopes, accelerometers, or even barometers) that support voice and data streaming. The memory subsystem 1110 may include memory and cache on the processor 1100, which may include L1 or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination, for example, for storing information and / or instructions for performing any of the operations described above or elsewhere herein. All or part of the memory and / or cache in the memory subsystem 1110 may be shared or used individually by any component or combination of components on the processor 1100 (e.g., GPU 1106, NPU 1104, and CPU 1102).

[0167] The Qualcomm AI Engine 1100 can be programmed and controlled using a software stack to perform some or all of the operations described herein, including, for example... A neural processing SDK is provided for inference on Android, Linux, and Windows. Developer libraries and services support programming languages, virtual platforms, and compilers. At lower levels of the software stack, system software includes a basic real-time operating system (RTOS), system interfaces, and drivers. The software stack supports various operating systems, including Android, Windows, Linux, and QNX, as well as deployment and monitoring infrastructures such as Prometheus, Kubernetes, and Docker. OpenCL and DirectML are supported for direct cross-platform access to the GPU 1106. For the CPU 1102, LLVM compiler infrastructure optimizations enable accelerated and efficient AI inference. Regarding the Qualcomm AI Engine 1100 and any of its components described above or elsewhere herein, one or more APIs described herein can, for example, be compiled into instructions that can be fetched by instruction fetching logic or equivalents, decoded by processor decoders or equivalents, scheduled (e.g., sequentially or out of order) for execution by a scheduler or equivalent, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. The API (and / or compiled instructions including the API) can be stored in any storage device (e.g., cache and / or memory) inside or outside the Qualcomm AI Engine 1100. The results of the API can be stored in storage devices inside or outside the Qualcomm AI Engine 1100, including registers, DRAM, flash memory, SRAM, cache, or other memory.

[0168] In at least one embodiment, processor 1100 or Qualcomm AI Engine 1100 may include one or more circuitry configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuitry, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0169] Figure 12AA processor 1200 according to at least one embodiment is illustrated. The processor 1200 may include a Scalable family processor from Intel Corporation, Santa Clara, California, or other processors sharing at least some of the components described herein. The processor 1200 may include one or more cores 1212(1)-1212(N) capable of performing the operations described elsewhere herein, where N is any integer greater than 1. Cores 1212(1)-1212(N) may be interconnected using ring and / or mesh interconnects. Utilizing a mesh interconnect architecture, arrays of vertical and horizontal communication paths may allow traversal from one core to another 1212(1)-1212(N) via the shortest path (jumping to the correct row along the vertical path and to the correct column along the horizontal path). For the mesh interconnect, a die may accommodate cores 1212(1)-1212(N) and may include a Converging Mesh Stop Point (CMS) grid that may be associated with cores 1212(1)-1212(N) (e.g., 1:1). Each core can be associated with a low-level cache (LLC) slice 1214(1)-1214(N), or cores 1212(1)-1212(N) can share a cache, such as a low-level cache. LLC 1214(1)-1214(N) can be inclusive or non-inclusive (having blocks not present in the high-level cache) by merging blocks from a higher-level cache (e.g., L2 cache). Each core and LLC slice can include a caching and home agent (CHA) (not shown), which can maintain cache consistency by providing resource scalability via mesh interconnect. Super Path Interconnect ( UPI 1216 provides cache coherency capabilities. UPI 1216 can provide coherent interconnects for scalable systems and allows multiple processors to share a single shared address space via links, such as, but not limited to, two or three UPI links per processor.

[0170] Processor 1200 may also include system agent 1210, which may house and / or perform various functions, such as, but not limited to, memory management, display functions, and / or input / output (I / O) functions. For example, processor 1200 may include one or more integrated memory controllers (IMCs) 1208. IMCs 1208 may control and manage memory, such as, but not limited to, different memory types, such as DDR RAM, such as DDR4, or other memory described elsewhere herein. System agent 1210 may include a display controller (not shown) for supporting one or more displays. System agent 1210 may also integrate PCIe 1204 (e.g., up to 20 PCIe lanes), which may, for example, be connected to an external dedicated graphics connector via a DMI bus (e.g., Intel's DMI 3.0 bus) 1206. System agent 1210 may include an image processing unit (IPU) (not shown) that integrates an on-die image signal processor (ISP). Structure 1202 provides scalability for connecting to other nodes (e.g., processors, such as processor 1200) and can, for example, be connected to Cornelis Networks (…). It can be used together with elements of a scalable system framework that provides performance for high-performance computing (HPC) workloads and the ability to scale to tens of thousands of nodes.

[0171] Figure 12B Components within a core 1212 according to at least one embodiment are illustrated. The core 1212 may include a front-end 1218, a back-end or execution engine 1232, and a memory subsystem 1242. The front-end 1218 may provide operations (e.g., operations described elsewhere herein) to the execution engine 1232 by decoding instructions stored in memory. For example, the front-end 1218 may include micro-operation (μOps) cache paths and / or traditional paths, and a branch prediction unit 1221 capable of determining path instructions. A traditional path of instructions may include fetching variable-length (e.g., x86) instructions from an L1 instruction cache 1220 and instruction fetch and pre-decode 1222, queuing these instructions into an instruction queue 1224, and decoding the instructions into μOps that can be provided to an allocation queue 1228 using a decoder 1226. Alternatively, the μOps cache path may include a cache that includes decoded μOps (μOps 1230) that can be sent to the allocation queue 1228. The allocation queue 1228 can act as an interface between the front end 1218 and the execution engine 1232, and can provide instructions to the execution engine 1232. For example, one or more APIs described herein can be compiled into instructions that can be stored, processed, and executed by the front end 1218 and the execution engine 1232, and stored in the memory subsystem 1242.

[0172] Execution engine 1232 can receive micro-operations into reordering buffer 1234, which can register, rename, and terminate μOPs. μOPs can be sent from the reordering buffer to scheduler 1236, which can be connected to one or more different execution units 1238, which can be connected to address generation units (AGUs) 1240. Execution units 1238 can perform operations such as basic arithmetic logic unit (ALU) operations, multiplication, division, and / or more complex operations, such as, but not limited to, various vector operations. Scheduler 1236 can manage the queuing of μOPs for one or more execution units 1238 based on, for example, the operations that need to be performed.

[0173] The memory subsystem 1242 can handle load and store requests as well as sorting operations. For example, μOPs may be associated with memory accesses (e.g., load and store), and these μOPs can be sent through dedicated scheduler ports that can perform these memory operations. For example, store and load operations can be sent to load and store buffers 1244. The memory subsystem 1242 may also include shared or separate L1 data and instruction caches 1246, and an L2 cache 1248 that can be used and shared by the L1 data and instruction caches 1246. (As described above regarding...) Figure 12A Each core 1212 can be connected to a slice of a level 3 cache (e.g., LLC1214), which can be shared by all cores 1212.

[0174] In at least one embodiment, processor 1200 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0175] Figure 13An AI accelerator 1300 according to at least one embodiment is illustrated. Processor 1300 may include a processor with an AI accelerator architecture manufactured by Intel Corporation, Santa Clara, California, or other processors sharing at least some of the components described herein. AI accelerator 1300 may use instructions that, if executed by AI accelerator 1300, cause AI accelerator 1300 to perform some or all of the processes and techniques described elsewhere herein. For example, with respect to AI accelerator 1300 and any components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic or equivalents, decoded by processor decoders or equivalents, scheduled (e.g., sequentially or out of order) for execution by a scheduler or equivalent, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device internal or external to AI accelerator 1300 (e.g., in cache and / or memory). The results of the API can be stored in internal or external storage devices of the AI ​​accelerator 1300, including registers, DRAM, flash memory, SRAM, cache, or other memory. The AI ​​accelerator 1300 may include one or more compute dies, which may include homogeneous or heterogeneous processors. The compute dies may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), or a combination of both.

[0176] In at least one embodiment, the computational die may include a computational engine for performing AI computations. In at least one embodiment, the computational die of the AI ​​accelerator 1300 may be split into any number (e.g., four) clusters, which may be referred to as DCORE (Deep Learning Core) 1306, and include any number of matrix multiplication engines (MME) 1308, tensor processor cores (TPC) 1310, memory management units 1312, and L2 caches 1314 in any combination. The MME 1308 may perform operations using matrix multiplication, such as fully connected layers, convolutions, and batch general matrix multiplication (GEMM). The MME 1308 may be equipped with a multiplication-accumulation unit (MAC) (not shown), which may perform general matrix multiplication (GEMM) operations, such as, but not limited to, AxB multiplication, which involves generating a tensor C [NxM] from two input tensors A [NxK] and B [KxN]. The MME 1308 may be programmed with array dimensions, positions, data types, and various operands. The MME 1308 can retrieve tensors A and B from memory and pull them into its streaming buffer for parallel matrix multiplication by the MAC. After completion, the MME 1308 can push tensor C back to memory. The TPC 1310 may include any number of scalar units for performing scalar operations, any number of vector units for performing vector operations, any number of register files or local memory units (e.g., vector local memory), and load and store components for instructions, which may be coupled to memory or caches (e.g., HBM, L3 cache, and / or L2 cache) (all not shown). The TPC can support different types of parallel processing, such as Very Long Instruction Word (VLIW) Single Instruction Multiple Data (SIMD) data types such as, but not limited to, FP32, BF16, FP16, and FP8 (both E4M3 and E5M2), UINT32, INT32, UINT16, INT16, UINT8, and INT8 data types. Any number of computational dies can be interconnected. Interconnects that can connect computing dies can be via intermediate bridges, for example, those intermediate bridges that are transparent to software.

[0177] The memory on the AI ​​accelerator 1300 may include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. The memory and / or cache system may be unified or separate. The compute die of the AI ​​accelerator 1300 may include on-chip memory comprising one or more levels (e.g., two levels) of cache. On-chip SRAM or other memory described elsewhere herein may be used as a unified last-level cache (L3) or split into multiple slices of L2 cache accessible to the MME 1308 and TPC 1310 groups. The use of on-chip memory as an L2 or L3 cache is entirely software-configurable, and the software can dynamically determine its optimal cache allocation based on I / O tensors. AI accelerator 1300 may include one or more memory management units (MMUs) 1322 for managing memory, such as allowing the AI ​​accelerator 1300 memory subsystem to run in virtual space when accessing VRAM.

[0178] AI accelerator 1300 may include a communication port (e.g., a PCIe Gen5 x16 port) 1302 for communicating with a host and scheduling and synchronization unit 1304. AI accelerator 1300 may include a media unit 1316, which may include any number or combination of media decoder engines (DECs) 1320 and rotation engines (ROTs) 1318. AI accelerator 1300 may include a network unit 1324, which may include any number or combination of network ports 1326 and an accompanying RDMA engine 1328, L2 cache, and memory (e.g., HBM2e or HBM3) stack. AI accelerator 1300 may include a programmable control path entity (not shown) for managing the parallel and efficient execution of the various engines. The control path may include a submission queue (SQ) that can be issued by the runtime system, a completion queue (CQ) that can be used for job completion reporting, a programmable scheduling mechanism that can be used for task scheduling, a programmable hardware synchronization mechanism or "synchronization manager (SM)" that can be used for hardware synchronization, and a programmable interrupt service mechanism or "interrupt manager (INTR)" that can pass asynchronous events to drivers.

[0179] AI Accelerator 1300 may include media decoding units supporting video formats such as, but not limited to, HEVC, Progressive H.264, SVC base layer, MVC, VP9, ​​JPEG, and Progressive JPEG. AI Accelerator 1300 may support post-processing of the decoded media stream, such as, but not limited to, image downsizing (image resizing), vertical and horizontal scaling at different scaling ratios, image enlargement, image cropping, bilinear scaling, and Lancos scaling. AI Accelerator 1300 may implement two post-processing channels per decoder unit, one for scalar (up and down) and the other solely for outputting the original image. AI Accelerator 1300 may include a hardware rotation engine that performs the following transformations on the input image: 2D rotation, 3D rotation, projection, image warping and de-warping, resampling of the input data at user-defined coordinates, and rescaling.

[0180] The RDMA 1328 based on converged Ethernet on the AI ​​accelerator 1300 enables scaling from a single node (i.e., from a single AI accelerator 1300 to hundreds or thousands of nodes or AI accelerators 1300). The network subsystem 1324 may include... The accelerator 1300 includes an In-Gigabit Ethernet Communication Library (IGCL), a master controller coordinating data movement, and a programmable scheduling mechanism that enables smooth engine activation while maintaining task dependencies. The accelerator network subsystem may include a Gigabit Ethernet NIC port 1326, a Layer 2 MAC (not shown), and an RDMA engine 1328. The AI ​​accelerator 1300 may include an aggregation engine for performing summation activities. All engines in the processor 1300 can run in parallel; for example, the MME 1308, TPC 1310, and NIC 1326 can all operate simultaneously. Dependencies may exist between operations running on different engines; for example, the output of one engine may be used as the input of another engine, and / or the MME, TPC, and NIC may be scheduled to run in parallel. When one engine completes its execution, another engine can be scheduled to begin working on the next operation (executed immediately after its input is ready).

[0181] The AI ​​accelerator 1300 can be operated and controlled using a software layer 1328, which may include low-level components such as, but not limited to, a graph compiler, an automatic kernel fusionist and pre-compiled kernel libraries, and integration with the AI ​​ecosystem such as, but not limited to, PyTorch, DeepSpeed, Hugging Face, vLLM, Ray, etc., or as described elsewhere in this document regarding software and programming platforms. The software layer 1328 may include implementations of algorithms such as, but not limited to, paged attention, flash attention, etc. The software layer 1328 may generate optimized binary code that implements a given model topology, such as, but not limited to, performing operator fusion, data layout management, parallelization, pipeline and memory management, and graph-level optimization.

[0182] In at least one embodiment, the AI ​​accelerator 1300 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0183] This paper describes a neuromorphic computing system employing a multi-core architecture, where each core houses computing elements including neurons, synapses with on-chip learning capabilities, and local memory for storing synaptic weights and routing tables. Figure 14 This is a simplified block diagram 1400 illustrating at least a portion of an example of such a neuromorphic computing device 1405 according to at least one embodiment. The neuromorphic computing device 1405 may include a neuromorphic processor from Intel Corporation, Santa Clara, California, or other processors that include at least a portion of the components described herein. As shown in this example, the device 1405 may be equipped with a network 1410 consisting of multiple neural network cores interconnected by a network on the device, thereby potentially defining multiple distinct connections between the cores. For example, the device 1405 may provide a network 1410 of spiking neural network cores, each core communicating via short packet pulse messages sent from one core to another through network channels. Each core (e.g., 1415) may have processing and memory resources, as well as logic, for implementing a number of primitive nonlinear time computation elements, such as, but not limited to, multiple (e.g., more than 1000) distinct artificial neurons (referred to herein as “neurons”). For example, each core may be able to implement multiple neurons concurrently, allowing the neuromorphic core to implement many, many neurons using the device 1405. With respect to neuromorphic computing device 1405 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to neuromorphic computing device 1405. The results of the APIs may be stored in storage devices internal or external to neuromorphic computing device 1405, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0184] continue Figure 14For example, neuromorphic computing device 1405 may also include processor 1420 and system memory 1425 for implementing one or more components to manage and provide the functionality of neuromorphic computing device 1405. For instance, a system manager 1430 may be provided to manage the global attributes and operations of neuromorphic computing device 1405 (e.g., attributes affecting core network 1410, multiple cores in network 1410, interconnection of neuromorphic computing device 1405 with other devices, managing access to global system memory 1425, and other potential examples). In one example, system manager 1430 may manage the definition and configuration of specific routing tables for individual routers in network 1410, orchestration of network definitions and attributes to be applied to network 1410 (e.g., weights, attenuation rates, etc.), core synchronization and time multiplexing management, routing input to appropriate cores, and other potential functions.

[0185] As another example, the neuromorphic computing device 1405 may also include a programming interface 1435 through which a user or system can specify the neural network definition to be applied (e.g., via routing tables and individual neuron attributes), implemented by the neuromorphic core grid 1410. A software-based programming tool may be provided with or separately from the neuromorphic computing device 1405, through which a user can provide a definition for a specific neural network to be implemented using the neuromorphic core network 1410. The programming interface 1435 can receive input from a programmer, then generate the corresponding routing table and populate the specified parameters into the local memory of each neuromorphic core (e.g., 1415) to implement the corresponding custom artificial neural network implemented by the neuromorphic core 1415.

[0186] In certain circumstances, neuromorphic computing device 1405 can advantageously engage and interoperate with other devices, including general-purpose computing devices, to enable specific applications and use cases. Therefore, external interface logic 1440 may be provided in certain situations to communicate with one or more other devices (e.g., via one or more defined communication protocols). External interface 1440 may be used to accept input data from another device or an external memory controller used as an input data source. External interface 1440 may additionally or alternatively be used to allow the results or outputs of computations performed using the neural network implemented using neuromorphic computing device 1405 to be provided to another device (e.g., another general-purpose processor implementing machine learning algorithms) to enable additional applications and enhancements, among other examples.

[0187] like Figure 14The diagram illustrates a network 1410 interconnected by a network on the device, showing a portion of a network structure interconnecting multiple neuromorphic cores (e.g., 1415a-d). For example, several neuromorphic cores (e.g., 1415a-d) can be provided in a mesh, each core interconnected via a network including multiple routers (e.g., 1450). In one implementation, each neuromorphic core (e.g., 1415a-d) can be connected to a single router (e.g., 1450) in the routers, and the router can be connected to at least one other router (e.g., [other router]). Figure 14 (As shown at 1410 in the diagram). As an example, in one particular implementation, four neuromorphic cores (e.g., 1415a-d) can be connected to a single router (e.g., 1450), and each router 1450 can be connected to two or more other routers to form a multi-core mesh, thereby allowing each neuromorphic core to interconnect with every other neuromorphic core in the neuromorphic computing device 1405. Furthermore, since each neuromorphic core can be configured to implement multiple different neurons, the router network of the neuromorphic computing device 1405 can similarly implement connections or artificial synapses (or simply "synapses") defined between any two of the potential many (e.g., 30,000+) neurons defined by the network definition using the neuromorphic cores 1410 provided in the neuromorphic computing device 1405.

[0188] Figure 14A block diagram of the internal components of an example implementation of the neuromorphic core 1415 is shown. In one example, a single neuromorphic core may implement a number of neurons (e.g., 1024) that share the architectural resources of the neuromorphic core 1415 in a time-division multiplexing manner. In one example, each neuromorphic core 1415 may include a processor block 1455 capable of executing arithmetic functions and routing related to the implementation of the digitally implemented artificial neurons, such as, but not limited to, those explained herein. Each neuromorphic core 1415 may also provide local memory in which routing tables of the neural network can be stored and accessed, accumulated potentials of each cell body of each neuron implemented using core 1415 can be tracked, parameters of each neuron implemented by core 1415 can be recorded, and other data and usage can be recorded. Components or architectural resources of the neuromorphic core 1415 may also include: an input interface 1465 for receiving input spike messages generated by other neurons on other neuromorphic cores; and an output interface 1470 for sending spike messages to other neuromorphic cores via a mesh network 1410. In some instances, the routing logic of the neuromorphic core 1415 can be implemented at least partially using the output interface 1470. Furthermore, in some cases, the core (e.g., 1415) can implement multiple neurons within an example SNN, and some of these neurons can be interconnected. In this case, spiking messages sent between neurons hosted on the core 1415 can forgo communication via the routing structure of the neuromorphic computing device 1405 and can instead be managed locally within the specific neuromorphic core 1415.

[0189] Each neuromorphic core may also include logic for implementing artificial dendrites 1480 and artificial cell bodies 1485 (hereinafter referred to as “dendrites” and “cell bodies”, respectively) for each neuron 1475. Dendrite 1480 may be a hardware-implemented process for receiving impulses from network 1410. Cell body 1485 may be a hardware-implemented process for receiving the current time-accumulated neurotransmitter mass of each dendrite and evolving the potential states of each dendrite and cell body to generate outgoing impulse messages at appropriate times. Dendrite 1480 may be defined for each connection receiving input from another source (e.g., another neuron). In one implementation, the dendritic process 1480 may receive and process the impulse message as it arrives serially from network 1410 in a time-division multiplexed manner. With the reception of impulses, neuronal activation (tracked using cell body 1485 (and local memory 1460)) may increase. When the activation of a neuron 1475 exceeds a threshold set for neuron 1475, neuron 1475 generates a spike message, which is propagated via output interface 1470 to a fixed set of fan-out neurons. The network distributes the spike messages to all destination neurons, which in turn can update their activation in a transient, time-dependent manner in response. This can lead to some of the destination neurons also exceeding their corresponding thresholds and triggering further spike messages, just as in real biological neural networks.

[0190] As described above, the neuromorphic computing device 1405 can reliably implement spiking-based neural computing models. Such models can also be called spiking neural networks (SNNs). In addition to neuronal and synaptic states, SNNs include temporal concepts. For example, in SNNs, communication occurs via event-driven action potentials or spiking, which convey no explicit information other than the spiking time and the implicit source and destination neuron pairs corresponding to the spiking transmission. The computation of the result of a dynamic nonlinear integral as a weighted spiking input occurs in each neuron. In some implementations, recurrent and dynamic feedback can be incorporated into the SNN computation model. Furthermore, various network connectivity models can be employed to model a wide range of real-world networks or relationships, including fully connected (all-to-all) networks, feedforward trees, completely random projections, "small-world" networks, and other examples. Isomorphic two-dimensional networks at the neuromorphic core (e.g., but not limited to...) Figure 14The network shown in the example can advantageously support all these network models. Since some or all of the cores of the neuromorphic computing device 1405 can be connected, some or all of the neurons defined in a core can also be fully connected via a certain number of router hops. The neuromorphic computing device 1405 may also include fully configurable routing tables for defining various neural networks by allowing neurons in each core to distribute their spurs to any number of cores in the grid 1410 to achieve a completely arbitrary connection graph.

[0191] In improved implementations of systems capable of supporting SNNs, for example, but not limited to... Figure 14 The example illustrates a very large-scale integrated circuit (VLSI) hardware device that can provide high-speed, reliable circuitry to implement SNNs (Spiritual Neural Networks) to model the information processing algorithms used by the brain, but in a more programmable way. For instance, while a biological brain can only perform a specific set of defined behaviors (a consequence of years of development), a neuromorphic processor device can provide the ability to rapidly reprogram all neural parameters. Therefore, a single neuromorphic processor can be used to implement a wider range of behaviors than a single slice of biological brain tissue. This distinction can be achieved by employing neuromorphic processors with neuromorphic designs that are radically different from those found in natural neural circuitry.

[0192] As an example, a neuromorphic processor can implement a spontaneous neural network (SNN) using time-multiplexed computation in both a spiking communication network and the neuronal mechanism of the neuromorphic computing device 1405. Therefore, the physical circuitry of the neuromorphic computing device 1405 can be shared by many neurons to achieve a higher neuron density. Through time multiplexing, the network can connect N cores with a total wiring length of O(N), while the length of discrete point-to-point wiring will be extended to O(N). 2 This significantly reduces wiring resources to accommodate planar and non-plastic VLSI routing techniques, among other examples. In the neuromorphic core, time multiplexing can be implemented through dense memory allocation, for example, using static random access memory (SRAM) with a shared bus, address decoding logic, and other multiplexed logic elements. The state of each neuron can be stored in the processor's memory, where data describing the state of each neuron includes the state of the collective synapse of each neuron, all currents and voltages on its membrane, and other example information (e.g., but not limited to configuration and other information).

[0193] Neuromorphic processors can be implemented in a “digital” manner, unlike other processors that employ more “analog” or “isomorphic” neuromorphic approaches. For example, a digital implementation can use digital adder and multiplier circuitry to integrate synaptic currents, in contrast to an analog isomorphic neuromorphic approach that accumulates charge on capacitors in a manner similar to how neurons accumulate synaptic charge on their lipid membranes. For instance, the accumulated synaptic charge for each neuron can be stored in the local memory of the corresponding core. Furthermore, at the architectural level of an example digital neuromorphic processor, reliable and deterministic operation can be achieved through time synchronization across the core network, ensuring that any two executions of the design, given the same initial conditions and configuration, will produce the same results. Asynchronicity can be reserved at the circuit level to allow individual cores to operate as quickly and freely as possible while maintaining determinism at the system level. Therefore, in neural computing, the concept of time as a time variable can be abstracted away from the “wall clock” time used by the hardware to perform computations. Thus, in some implementations, a time synchronization mechanism can be provided that globally synchronizes the neuromorphic cores at discrete time intervals. The synchronization mechanism allows neural computation to be completed at the fastest speed allowed by the circuit, and there is a difference between the runtime and the biological time for modeling neuromorphic systems.

[0194] In operation, the neuromorphic computing device 1405 can start in an idle state when all neuromorphic cores are inactive. As each core asynchronously loops through its neurons, it generates impulse messages, which are routed by the mesh interconnect to the appropriate destination core containing all destination neurons. The implementation of multiple neurons on a single neuromorphic core can be time-multiplexed, and time steps can be defined, where all impulses involving multiple neurons can be processed and considered using the shared resources of the respective cores. When each core completes its service to its neurons within the corresponding time step, in some implementations, the core can communicate with neighboring cores using synchronization messages (e.g., using a handshake) to refresh the mesh of all transmitted impulse messages, allowing the core to safely determine that all impulses have been serviced within a certain time step. At this point, all cores can be considered synchronized, allowing them to advance their time steps and return to the initial state to begin the next time step.

[0195] Given this context, as described above, a device (e.g., 1405) can be provided to realize an interconnected neuromorphic core network 1410, wherein core 1415 can realize multiple artificial neurons capable of interconnecting to realize an SNN. Each neuromorphic core (e.g., 1415) can provide two loosely coupled asynchronous processes: an input dendrite process (e.g., 1480) that receives impulses from network 1410 and applies them to the appropriate destination dendritic compartment at an appropriate future time; and an output cell body process (e.g., 1485) that receives the current-time accumulated neurotransmitter mass of each dendritic compartment and evolves the membrane potential state of each dendrite and cell body to generate an outgoing impulse message at an appropriate time (e.g., when the threshold potential of the cell body is reached). It should be noted that, from a biological perspective, the names of dendrites and cell bodies used herein are only approximate to the function of these features and should not be interpreted too literally.

[0196] In at least one embodiment, the neuromorphic computing device 1405 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0197] Figure 15 This is a block diagram of an embodiment of a multi-node network capable of enabling remote memory computing according to any embodiment. System 1500 may represent the node network described herein, for example, the node network may be used to perform some or all of the operations described herein. System 1500 may represent a data center. System 1500 may represent a server farm. System 1500 may represent a data cloud or processing cloud. System 1500 may represent a supercomputer. System 1500 may include tens, hundreds or thousands of nodes. The nodes of System 1500 may include processors, such as, but not limited to, a central processing unit (CPU), a graphics processing unit (GPU), or any combination of processors described herein, such as, but not limited to, processors described herein. Figure 9-21B Other processors in the system. For any processor in System 1500 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) inside or outside the processor or node. The results of APIs may be stored in storage devices inside or outside the processor or node, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents. System 1500 may include more than nine thousand nodes, each node comprising two Intel Xeon Max processors, six Intel Max series GPUs, and a unified memory architecture, such as, but not limited to, the architecture used in Intel's Aurora supercomputer in Santa Clara, California, or other supercomputers that share at least some of the components described herein.

[0198] One or more clients 1502 send requests to system 1500 via network 1504. Network 1504 represents one or more local area networks, wide area networks, or a combination of both. Clients 1502 can be human or machine clients that generate requests for operations to be performed by system 1500. System 1500 executes the application or data computation task requested by client 1502.

[0199] System 1500 may include one or more racks, which represent structural and interconnect resources for housing and interconnecting multiple computing nodes. Rack 1510 may include multiple nodes 1530. Rack 1510 may carry multiple blade assemblies 1520(0)-1520(N-1), where N is an integer greater than or equal to 2. Carrying may refer to providing power, structural or mechanical support, and interconnection. Blades 1520(0)-1520(N-1) may refer to computing resources on a printed circuit board (PCB), where the PCB houses the hardware components of one or more nodes 1530. Blades 1520(0)-1520(N-1) may include or exclude a chassis, enclosure, or other “box” besides those provided by rack 1510. Blades 1520(0)-1520(N-1) may include an enclosure with exposed connectors for connection to rack 1510. System 1500 may or may not include rack 1510, and each blade (e.g., 1520(0)) may include a chassis or housing that may be stacked or otherwise closely proximate with other blades and allow nodes 1530 to interconnect. System 1500 may include 10,624 compute blades, comprising 63,744 Intel Max series GPUs and 21,248 Intel Xeon Max CPUs on 166 racks.

[0200] System 1500 may include architecture 1570, which represents one or more interconnectors of nodes 1530. Architecture 1570 may include multiple switches 1572 or routers or other hardware for routing signaling between nodes 1530. Furthermore, architecture 1570 may couple system 1500 to network 1504 for access by client 1502. In addition to routing devices, architecture 1570 may also be considered to include cables or ports or other hardware devices for coupling nodes 1530 together. Architecture 1570 may have one or more associated protocols for managing signaling routing through system 1500. One or more protocols are at least partially dependent on the hardware devices used in system 1500.

[0201] As shown in the figure, rack 1510 may include N blades (e.g., 1520(0)-1520(N-1)). In addition to rack 1510, system 1500 may also include rack 1550. As shown in the figure, rack 1550 may include M blades (e.g., 1560(0)-1560(M-1)). M is not necessarily the same as N; therefore, it is understood that various different hardware device components may be used and coupled together into system 1500 via structure 1570. Blades 1560(0)-1560(M-1) may be the same as or similar to blades 1520(0)-1520(N-1). Node 1530 may be any type of node described herein and is not necessarily of the same type. System 1500 is not limited to homogeneous or non-homogeneous systems.

[0202] The nodes in blade 1520(0) are shown in detail. However, other nodes in system 1500 may be the same or similar. At least some nodes 1530 may be compute nodes, having processor 1532 and memory 1540. A compute node is a node having processing resources (e.g., one or more processors) that executes an operating system and can receive and process one or more tasks. At least some nodes 1530 may include storage server nodes, which have servers as processing resources 1532 and memory 1540. A storage server is a node having more storage resources than a compute node, and instead of having processors for performing tasks, a storage server includes processing resources for managing access to storage nodes within the storage server.

[0203] Node 1530 may include interface controller 1534, which may represent logic for controlling node 1530's access to structure 1570. The logic may include hardware resources for interconnecting to physical interconnect hardware. The logic may include software or firmware logic for managing the interconnect. Interface controller 1534 may include a host structure interface, which may include a structure interface according to any embodiment described herein.

[0204] Node 1530 may include a memory subsystem 1540. Memory 1540 may include a memory computation resource (comp) 1542, which represents the ability of memory 1540 to perform one or more memory computations. System 1500 supports remote memory operations, such as, but not limited to, those described elsewhere herein. Therefore, node 1530 may request a remote node to perform a memory computation, wherein the data used for the computation remains local to the executing node and is not sent via structure 1570 or from memory to the structure interface. In response to the execution of the memory computation, the executing node may provide the result to the requesting node.

[0205] Processor 1532 may include one or more individual processors. Each individual processor may include a single processing unit, a multi-core processing unit, or a combination thereof. A processing unit may include a main processor, such as, but not limited to, a CPU (Central Processing Unit), a peripheral processor (such as, but not limited to, a GPU (Graphics Processing Unit)), or a combination thereof. Memory 1540 may be or include a memory device and a memory controller.

[0206] The term "memory device" can refer to different types of memory. Memory devices generally refer to volatile memory technology. Volatile memory is memory whose state (and the data stored within it) is uncertain if power is interrupted. Non-volatile memory is memory whose state is deterministic even if power is interrupted. Dynamically volatile memory can refresh the data stored in the device to maintain its state. An example of dynamically volatile memory includes DRAM (Dynamic Random Access Memory) or variations thereof, such as, but not limited to, Synchronous DRAM (SDRAM). The memory subsystem described in this article is compatible with a variety of memory technologies, such as, but not limited to, DDR3 (Double Data Rate version 3, originally released by JEDEC (Joint Electron Device Engineering Committee) on June 27, 2007, currently version 21), DDR4 (DDR version 4, initial specification released by JEDEC in September 2012), DDR4E (DDR version 4, extended version, currently under discussion by JEDEC), LPDDR3 (Low Power DDR version 3, JESD209-3B, released by JEDEC in August 2013), and LPDDR4 (Low Power Double Data Rate (LPDDR) version 4). JESD209-4 (originally released by JEDEC in August 2014), WIO2 (Wide I / O2), JESD229-2 (originally released by JEDEC in August 2014), HBM (High Bandwidth DRAM), JESD235 (originally released by JEDEC in October 2013), DDR5 (DDR version 5, currently under discussion by JEDEC), LPDDR5 (currently under discussion by JEDEC), HBM2 (HBM version 2, currently under discussion by JEDEC) or combinations of other memory technologies, as well as technologies derived from or extended based on such specifications.

[0207] In addition to or as an alternative to volatile memory, in one embodiment, a reference to a memory device may refer to a non-volatile memory device whose state is deterministic even when power is interrupted. In one embodiment, a non-volatile memory device is a block-addressable memory device, such as, but not limited to, NAND or NOR technology. Therefore, the memory device may also include future-generation non-volatile devices, such as, but not limited to, three-dimensional cross-point (3DXP) memory devices, other byte-addressable non-volatile memory devices, or memory devices using chalcogenide phase change materials (e.g., chalcogenide glasses). In one embodiment, the memory device may be or include multi-threshold NAND flash memory, NOR flash memory, single-level or multi-level phase change memory (PCM) or switched phase change memory (PCMS), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), magnetoresistive random access memory (MRAM) incorporating memristor technology, or spin-transfer torque (STT)-MRAM, or any combination of the foregoing, or other memories.

[0208] In at least one embodiment, system 1500 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0209] Figure 16An accelerated processing unit 1600 according to at least one embodiment is illustrated. The accelerated processing unit 1600 may include a processor based on the CDNA architecture of AMD Inc., Santa Clara, California, or other processors sharing at least some of the components described herein. The accelerated processing unit 1600 may include one or more accelerator complex dies (XCDs) 1604 for performing the operations described elsewhere in this document, such as, but not limited to, graphics processing and / or parallel processing and instruction-level parallel computing, including support for multiple precisions (INT8, FP8, BF16, FP16, TF32, FP32, and FP64) and sparse matrix data (i.e., sparsity). In some cases, the XCD may be referred to as a graphics computing die (GCD). The accelerated processing unit 1600 may include one or more complex computing dies (CCDs) 1606 for performing the operations described elsewhere in this document, such as, but not limited to, operations performed by a host processor. In some cases, the CCD may be referred to as a core complex or CCX, such as, but not limited to, the CCX used in AMD Ryzen processors. XCDs and CCDs can share any type of cache or memory (e.g., one or more memory cells 1602), or a cache or memory can be allocated to each XCD or CCD or group of XCDs or CCDs. For example, AMD Infinity Fabric within the package connects XCDs and CCDs to a shared AMD Infinity Cache 1608, and in some embodiments, to high-bandwidth memory (e.g., HMB3). Accelerated processing unit 1600 may include an AMD MI300a processor comprising three CPU die (or CCD) and six accelerator die (XCD) on top of four input-output dies (IODs) that may be layered on a single silicon die (e.g., via AMD Infinity Fabric) and linked together to eight high-bandwidth DRAM stacks in a ring to form a superchip. For systems using only accelerators, the AMD MI300x processor replaces the CCD with two or more XCDs.

[0210] Accelerated processing unit 1600 may include one or more input / output (I / O) interfaces. For example, XCD 1604 and CCD 1606 may coexist on one or more input-output dies (IODs) 1610, which may include one or more I / O interfaces. IOD 1610 may include any number and type of I / O interfaces (e.g., PCI, PCI expansion (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices may be coupled to I / O interfaces 1670. The I / O interfaces of IOD 1610 may also be used to connect one or more accelerated processing units 1600, for example, in a server architecture.

[0211] Accelerated processing unit 1600 may include one or more memory units 1602 for storing instructions and other information for performing the operations described in other parts of this document. Memory units 1602 may include any volatile memory, such as, but not limited to, the memory types described in other parts of this document, and may include, for example, high-bandwidth memory (e.g., HMB3) or high-bandwidth DRAM. The memory associated with accelerated processing unit 1600 (e.g., memory unit 1602) may include system memory, which can be used for, for example, commands, instructions, and constants, as well as input and output. Memory unit 1602 may also include device memory, which can be used for storage and, for example, for commands, instructions, and constants, as well as input and output, as a return buffer, and for private data. Memory unit 1602 may be linked to one or more IODs 1610. In at least one embodiment, L1 cache 1620 initiates a memory hierarchy including a shared L2 cache 1628 (e.g., within an XCD). AMD Infinity Cache TM It is the last-level cache (LLC) located on the active I / O die (IOD). The CCD1606 and XCD1604 can have independent or shared memory. AMD Infinity architecture and AMD Infinity Fabric TM The technology enables consistent, high-throughput unification of GPU and CPU chip technologies (such as XCD, CCD, and / or CCX) with memory (such as stacked HBM3 memory) in a single device and across multiple device platforms.

[0212] like Figure 16As shown, the XCD 1604 may include a set of shared global resources 1630, which may include a hardware scheduler 1632 and an asynchronous compute engine (ACE) 1624. The ACE 1624 sends tasks (e.g., compute shader workgroups) to compute units (CUs or cores) 1634. Each of the ACEs 1624 (e.g., four) may be associated with a CU 1634 (e.g., 40 CUs), and some CUs 1634 may be disabled for yield management. CUs 1634 may have dedicated caches or shared caches (e.g., L2 caches) 1628 for consolidating all memory traffic on a single die. CU1634 may include threaded and parallel processor cores, including instruction fetching and scheduling using a scheduler (S) 1612, a matrix core unit (MCU) 1616, and a shader core (SC) 1618 (e.g., execution units for scalar, vector, and matrix data types), and a load / store pipeline with an L1 cache 1620 and a local data share (LDS) 1614. The local data share may include, for example, sticky-note RAM with built-in arithmetic capabilities, allowing data sharing between threads in a workgroup. Instruction cache 1640 (e.g., for storing and providing instructions for performing operations described elsewhere in this document) and constant cache 1638 may be connected to one or more CUs and may be shared between two CUs. Matrix core 1616 can handle various data types, such as, but not limited to, INT8, FP8, FP16, BF16, and TF32 data types. Accelerated processing unit 1600 may include computation units 1634, which may be arranged in an array format, such as as a data parallel processor (DPP) array. The hyper-threaded dispatch processor 1642 can communicate with the compute unit 1634, and the command processor 1644 can read commands written by the host to memory-mapped registers in the system memory address space (not shown). When a command is completed, the command processor 1644 can send a hardware-generated interrupt to the host processor (e.g., a CCD). The memory controller 1636 can also directly access all device memories and system memory regions specified by the host. To satisfy read and write requests, the memory controller 1636 can perform the functions of a direct memory access (DMA) controller, including calculating the memory address offset based on the format of the requested data in memory. For example, one or more APIs described herein can be compiled into instructions that can be stored in the instruction cache 1640, then fetched by the instruction fetch logic in the processor 1640, decoded by the processor decoder or equivalent, scheduled (e.g., sequentially or out of order) by the scheduler or equivalent for execution, executed by the execution logic or equivalent, reordered, and then retired by the retirement logic or equivalent.The API (and / or compiled instructions including the API) can be stored in any storage device, either inside or outside the processor 1600 (e.g., in a cache and / or memory). The results of the API can be stored in storage devices, either inside or outside the processor 1600, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0213] Applications may include programs running on the main processor (e.g., a CCD) and programs running on one or more XCDs (referred to as kernels). Programs can be controlled by host commands that set internal base addresses and other configuration registers, specify data fields on which the accelerator processing unit 1600 can run, invalidate and flush caches on the accelerator processing unit 1600, and cause the accelerator processing unit 1600 to begin executing a program. A kernel can be referred to as a program executed by the accelerator processing unit 1600. Kernels can execute independently on each work item or as a group of work items, referred to as a wavefront, which can execute kernels on all (e.g., 64) work items in a single pass. The computation unit 1634 may include: a scalar arithmetic logic unit (ALU) that can operate on a single value for each wavefront (shared by all work items); a vector ALU that can operate on a unique value for each work item; a local data share 1614 that allows work items within a workgroup to communicate and share data; a scalar memory (not shown) that can transfer data between the scalar general-purpose registers (SGPRs) and memory via cache; and a vector memory that can transfer data between the vector general-purpose registers (VGPRs) and memory, including sampling of texture maps. Kernel control flow can be manipulated using scalar ALU instructions, which may include if / else statements, branches, and loops. Scalar ALU (SALU) and memory instructions can operate on the entire wavefront and operate on one or more SGPRs. Vector memory and ALU instructions can operate on all work items in the wavefront simultaneously.

[0214] In at least one embodiment, the accelerated processing unit 1600 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0215] Figure 17Processor 1700 is shown, including, but not limited to, a Zen architecture-based processor (e.g., Zen1, 2, 3, 4, 5, or other architectures) from AMD Inc. of Santa Clara, California, or other processors that share at least some of the components described herein. Processor 1700 includes one or more CPU dies 1702(1)-1702(N), where N is any integer greater than 1. CPU die 1702 may include any number of processor cores 1716 (e.g., for performing any operations described elsewhere herein) and any number of cache memories (e.g., for storing instructions and other information to perform any operations described elsewhere herein). For example, L2 cache unit 1718 may be coupled to processor core 1716, and processor core 1716 may share and / or be individually coupled to L2 cache unit 1718. Processor core 1716 can be coupled to L3 cache 1722 and / or a shared L3 cache, which can be the lowest level cache (LLC) 1722 used to access data and other information used by processor core 1716. One or more processor cores 1716 and one or more L2 cache units 1718 can be included in a core complex (CCX) 1720, which can include (e.g., 32MB) a shared cache (e.g., L3 cache 1722). Core complex 1720 can be manufactured onto a die (CCD or CPU die) 1702. For example, up to 12 core complexes 1720 can be configured in a processor along with 8 CPU dies 1702, providing up to 96 processor cores 1716 for processor 1700. For example, a "Zen 4c" core complex 1720 can include up to 8 cores 1716 and a shared 16MB L3 cache 1722. Two core complexes from these core complexes 1720 can be combined onto a single CPU die 1702, resulting in 16 cores per die and a total of 32MB of L3 cache per die 1722. Up to eight CPU dies 1702 can be combined with I / O units 1704 to provide up to 128 processor cores 1716 for the CPU. Up to four "Zen 4c" dies mentioned above can be combined to provide up to 64 processor cores 1716 for the CPU.

[0216] Processor 1700 may include various configurations for input / output operations, which will be further described herein. I / O unit 1704 may include one or more memory controllers 1706 capable of managing the memory usage of processor 1700 (e.g., DDR5 memory). I / O unit 1704 may include one or more SATA disk controllers for managing storage device 1712, and one or more Compute Express Links (CXL) for providing CPU-to-device and CPU-to-memory connectivity, which can be flexibly assigned to specific functions during server design. TM 1.1+ Memory controller 1714. I / O unit 1704 may include PCIe controller 1708 for connecting peripherals and other components connected to processor 1700. I / O unit 1704 may also include USB port 1710 for connecting to other components separate from processor 1700. CPU die 1702 may support any number of connections to I / O unit 1704, for example, one or two connections. As shown, I / O unit 1704 may include components further described herein, and I / O unit 1704 may be an I / O die accommodating several different components. Memory controller 1706, PCIe controller 1708, USB port 1710, SATA controller 1712 and / or CXL controller 1714 may be individually integrated into any location within processor 1700, or integrated in any group or combination.

[0217] Processor 1700 may include an Infinity Fabric 1724 interconnect (which may be similar to or based on a PCIe architecture) that provides connectivity between the CPU (e.g., CPU die 1702(1)-1702(N)), graphics processor 1726, inference engine 1732, and other components in a multi-chip architecture (e.g., security processor 1728 and I / O unit 1704). One or more AMD Infinity Fabrics TM Interconnect 1710 can be connected to CPU dies 1702(1)-1702(N) and used as a connection between CPUs. One or more Infinity Fabric connections 1710 can connect each CPU die 1702 to the I / O unit 1710.

[0218] In at least one embodiment, processor 1700 may include a central processing unit (CPU) and other related hardware and software described above and further herein. Processor 1700 may also include a graphics processor 1726. Graphics processor 1726 may be used for image generation and processing, as well as other computations and operations described further herein. Graphics processor 1726 may be based on AMD's RDNA 3 or 3.5 architecture, located in Santa Clara, California. Graphics processor 1726 may include a graphics computing die (GCD) and a memory cache die (MCD). GCD may include any number of computing units (CUs) for graphics or other processing, such as operations performed by an arithmetic logic unit (ALU) described further herein. Graphics processor 1726 may include an L2 cache available for use by the computing units. MCD (not shown) may include any number of memory cells and may include a cache (e.g., an L3 cache) and a memory interface for coupling to memory (e.g., memory 1742(1)-(N), where N is an integer). Components within the graphics processor 1726 can be connected using various methods, such as using Infinity Fabric 1724 interconnects, either internally or externally to the graphics processor 1726.

[0219] Inference engine 1732 can provide neural processing capabilities to processor 1700 for computational processes used in neural networks, deep learning, and other AI-related operations described further herein. Processor 1700 may include: a security processor 1728 for managing the security of processor 1700; a display controller 1730 for controlling the display; a system management unit 1734 for managing and operating some or all components on processor 1700; a multimedia engine 1736 for audio and video operations; a fusion controller hub 1738 for managing USB, SATA, and PCIe connections to the processor; and a sensor fusion hub 1740 for managing sensors (e.g., accelerometers). Processor 1700 may also include memory 1742(1)-(N), where N is any integer. Memory may include different memory types, such as LPDDR5 and / or DDR5, or other memory described elsewhere herein.

[0220] To perform the operations further described herein, processor 1700 may include an execution pipeline including a front end that may include a cache for storing instructions (e.g., an L1 cache) (not shown). A branch predictor may modify the instruction stream. Instructions may be decoded by a decoder, dispatched to a back end for execution, and renamed. For example, the instruction fetch and decode pipeline may be dispatched to integer or floating-point execution operations, which may be scheduled by a scheduler and passed to vectors and / or general-purpose registers. Floating-point multipliers and / or addition operations may be processed, and an arithmetic logic unit (ALU) may also be used to perform computations, such as arithmetic and logical operations. The output of the computation unit may be coupled to a load / store queue that may be connected to a cache, such as an L1 cache and / or an L2 cache.

[0221] With respect to processor 1700 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents (e.g., AVX-512 instructions based on a SIMD model). These instructions or equivalents may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to processor 1700. The results of the APIs may then be stored in storage devices internal or external to processor 1700, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0222] In at least one embodiment, processor 1700 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0223] Figure 18 An example of a processing core 1800 is shown, which may implement an Arm architecture (e.g., v9.0-A) or other processors that share at least some of the components described herein. Neoverse TM The V2 core 1800 can be implemented within a DynamIQ shared unit (DSU) cluster via a DSU-110 interconnect 1854 for connecting one or more cores, for example, for parallel processing. Neoverse TM The V2 core can be implemented as a single core in a DSU cluster configured for direct interconnection, with or without L3 cache, listener filter, or listener control unit (SCU) logic (not shown). Neoverse TMThe V2 core may include a CPU bridge 1852 for connecting core 1800 to the DSU-110 interconnect. This bridge may also connect core 1800 to an external memory system and the remainder of the on-chip system. The L1 instruction memory system 1802 may fetch instructions from instruction cache 1804 and deliver instructions (e.g., one or more compileable APIs described herein) to instruction decoding unit 1810, for example, to perform some or all of the operations described above or elsewhere herein. The L1 instruction memory system 1802 may include L1 instruction cache 1804 (e.g., with 64-byte cache lines), L1 instruction translation back buffer (TLB) 1806 (e.g., natively supporting 4KB, 16KB, 64KB, and 2MB page sizes), and macro operation cache (MOP) 1808 (e.g., a 1536-entry, 4-way skewed associated L0 MOP cache), which may include decoded and optimized instructions for higher performance. Instruction decoding unit 1810 may decode AArch64 instructions into its internal format. The register renaming unit 1812 can perform register renaming to facilitate out-of-order execution and dispatch decoded instructions to various issue queues. The instruction issuing unit 1814 can control when decoded instructions are dispatched to the execution pipeline, and it can include an issue queue for storing instructions to be dispatched to the execution pipeline. The integer execution pipeline 1816 can be included in the execution pipeline and includes an integer execution unit 1818 that can perform arithmetic and logical data processing operations. The vector execution unit 1820 can be included in the execution pipeline and can execute advanced SIMD and floating-point arithmetic (FPU) 1822, execute Scalable Vector Extension (SVE) and Scalable Vector Extension 2 (SVE2) instructions 1824, and can also selectively execute cryptographic instructions 1826. The advanced SIMD can include a media and signal processing architecture that primarily adds instructions for audio, video, 3D graphics, image, and speech processing. The floating-point architecture provides support for single-precision and double-precision floating-point operations. The L1 data memory system 1830 executes load and store instructions, as well as service memory coherence requests. The L1 data memory system 1830 may include an L1 data cache 1832 and a fully associative L1 data TLB 1834, natively supporting 4KB, 16KB, and 64KB page sizes and 2MB and 512MB block sizes. The memory management unit (MMU) 1828 provides fine-grained memory system control through a set of virtual-to-physical address mappings and memory attributes, which are stored in a translation table and saved to the TLB 1834 after address translation. The L2 memory system 1836 may include an L2 cache 1838 and can be connected to the DSU-110 1854 via an asynchronous CPU bridge 1852. TMThe V2 core 1800 supports a range of debugging, testing, and tracing options, including the tracing unit 1842, the tracing buffer 1840, and the embedded logic analyzer (ELA) 1848. Neoverse TM The V2 core 1800 implements the Statistical Analysis Extension (SPE) 1844, which provides a statistical view of the performance characteristics of executed instructions. Software writers can leverage these views to optimize code for better performance. The Performance Monitoring Unit (PMU) 1846 provides a performance monitor that can be configured to collect statistics on the operation of each core and memory system. This information can be used for debugging and code analysis. The General Purpose Interrupt Controller (GIC) CPU interface 1850, when integrated with external allocator components, serves as a resource for supporting and managing interrupts in a cluster system. In a cluster, each Neoverse... TM There can be a CPU bridge 1852 between the V2 core 1800 and the DSU-110 1854. The CPU bridge 1852 can control the buffering and synchronization between the core 1800 and the DSU-110 1854. The CPU bridge 1852 can be asynchronous to allow each core 1800 to use a different frequency, power, and area implementation point. The CPU bridge 1852 can operate synchronously without affecting other interfaces, such as, but not limited to, asynchronous debug and tracing interfaces.

[0224] In at least one embodiment, core 1800 may include one or more circuitry configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuitry, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0225] Figure 19 One or more chips including one or more tensor processing units (TPUs) 1900 are shown according to at least one embodiment. Figure 19 The TPU 1900 may include an application-specific integrated circuit (ASIC), for example, for performing some or all of the operations described above or elsewhere herein, such as, but not limited to, machine learning workloads that accelerate the execution of matrix operations. The TPU 1900 may be an ASIC from Alphabet Corporation, Mountain View, California. The cloud TPU includes a cloud service that enables the TPU to be used as a scalable resource for processing tasks, such as, but not limited to, machine learning workloads that can run on frameworks such as, but not limited to, TensorFlow, PyTorch, and JAX.

[0226] Chip 1900 may include any number of TPUs, which may include a tensor core 1906. Tensor core 1906 may include one or more core sequencers 1908, vector processing units (VPUs) 1910, matrix multiplication units (MXUs) 1912(A)-1914(N) (where N is any integer greater than 1), and transpose permutation units 1916. Core sequencer 1908 may fetch instructions (e.g., VLIW (Very Long Instruction Word)) from the instruction memory (Imem) of core 1906, perform scalar operations using scalar data memory (Smem) and scalar registers (Sregs) (not shown), and forward vector instructions to the vector processing units (VPUs) 1910. For example, an instruction may initiate eight operations: two scalar operations, two vector ALU operations, vector loading and storing, and queuing data into the matrix multiplication and transpose units and a pair of slots for queuing data from them. The VPU1910 can perform vector operations using a large on-chip vector memory (Vmem) and vector registers (Vregs). The VPU1910 can stream data to or from the MXU via a decoupled FIFO. The VPU1910 can collect and distribute data to the Vmem using both data-level parallelism (2D matrix and vector function units) and instruction-level parallelism (8 operations per instruction). Large two-dimensional matrix multiplication units (MXUs) 1912(A)–1912(N) can, for example, use a systolic array to reduce area and power consumption, and use a large, software-controlled on-chip memory instead of a cache. The transpose-reduction-permute unit 1916 can perform matrix transpose, reduction, and permute operations (e.g., 128x128) on the VPU1910 channels. A high-bandwidth memory 1904 can be used for on-chip applications and can be coupled to the host queue 1902, for example, via PCIe. One or more chips 1900 can be connected together for computing. For example, one or more chips 1900 can be connected to form a torus, such as a 2D torus. Chips 1900 can also include any number (e.g., four) of inter-core interconnect (ICI) links 1918, which can enable direct connections between chips to form a supercomputer.

[0227] For any processor in chip 1900 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) external to or internal to any processor in chip 1900. The results of the APIs may then be stored in any storage device internal to or external to any processor in chip 1900, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0228] In at least one embodiment, chip 1900 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0229] Figure 20 A vector processor according to at least one embodiment is illustrated. The vector processor 2000 may support the RISC-V standard. The vector processor 2000 may include one or more cores 2010 (e.g., scalar units) and one or more vector processing units (VPUs) 2042 (e.g., vector units), which may, for example, perform some or all of the operations described above or elsewhere herein. The core 2010 may include an Andes Custom Extension (ACE) 2016, which can be used, for example, to deliver custom instructions to the processor 2000 via an ACP 2038. The core 2010 may include a 1-cycle multiplier and a 1-cycle instruction / data local memory (ILM / DLM) for improving parallelism by allowing simultaneous instruction fetching and data access. A memory management unit (MMU) 2024 manages system memory and cache, and provides branch execution, instruction pair issuance, L1 instruction / data cache, and local memory storage. The core 2010 may include a physical memory protection and programmable physical memory attribute unit (PMP / PPMA) 2022. Core 2010 may include a digital signal processor (DSP) 2028 and a floating-point unit (FPU) 2026, as well as a load-memory unit (LSU) 2032 for interaction with memory hierarchies (D$2034 and I$2030). Core 2010 may include a branch prediction unit 2018 and a multiplier unit 2020.

[0230] The vector processing unit (VPU) 2042 may include one or more vector function units (FU) 2046(A)-2046(N) (these vector function units may be linked together for parallel processing), a separate memory path for loading / storing RISC-V vectors (RVV) via ACE-RVV 2048 and AndesStreaming port (ASP) 2044, and a vector load / store unit (VLSU) 2050.

[0231] The vector processor 2000 may include bus interfaces, such as, but not limited to, a cacheable L2 cache port 2056, a non-cacheable MMIO port 2054, an input-output coherence port (IOCP) 2058 for a cacheless bus master, a local memory access port for accessing the ILM / DLM 2012 (which can be coupled to SRAM 2006) and the high-bandwidth vector memory (HVM) 2036, and a shared peripheral port (SPP) 2052 for external peripherals. Other memory ports include the LM slave port AXI 2002, the HVM subordinate port AXI 2004, the MEM (AXI) 2062, and the AXI 2060. Trace I / F 2014 can be captured, encoded, and transmitted off-chip via Inst. Trace I / F 2008 (e.g., a record of executed processor instructions). Software tools can use Inst. Trace I / F 2008 to reconstruct the exact execution sequence of a program.

[0232] For any processor in the processor 2000 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. The API (and / or compiled instructions including the API) may be stored in any storage device external to or internal to the processor 2000 (e.g., in cache and / or memory). The results of the API may then be stored in storage devices internal to or external to the processor 2000, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0233] In at least one embodiment, the vector processor 2000 may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0234] Figure 21A A schematic diagram of an example multi-core tiled processor microarchitecture is shown. Figure 21A Multi-core tiled processors in a system can include language processing processors. For example... Figure 21AAs shown, each “tile” in the processor architecture is a processing element bundled together using an on-chip network (NoC), which can be used to perform some or all of the operations described above or elsewhere in this document. For example, each tile may have an instruction dispatch 2104 and integer (INT) units 2106 and floating-point (FP) units 2108, a load-memory unit (LSU) 2112 for engaging with a memory hierarchy (data cache (D$) 2110 and instruction cache (I$) 2114), and a network (NET) interface 2116 for communicating with other tiles. Some tiles in processor 2100 may include a memory controller 2102 for managing and controlling memory, as further described herein. Processor 2100 may have a functionally sliced ​​architecture. Processor 2100 may reside on an application-specific integrated circuit (ASIC). Figure 21A The layout of an ASIC can be represented. Processor 2100 may include a coprocessor designed to execute instructions for a predictive model. A predictive model refers to any model configured to make predictions based on input data. The predictive model can use a classifier for classification predictions. The predictive model can be a machine learning model, such as, but not limited to, a tensor flow model, and processor 2100 is a tensor flow processor.

[0235] The processor 2100 can employ different microarchitectures, which will Figure 21B Each tile in the process represents a separate functional unit. Conversely, the functional tiles 2124 of the processor 2100 can be aggregated into multiple functional processing units (hereinafter referred to as "slices") 2104, each corresponding to a specific functional type (e.g., FP / INT 2118, NET 2120, MEM 2122). For example, as... Figure 21B As shown, each slice may correspond to a row of functional tiles extending in a north-south direction. Furthermore, the processor 2100 may also include communication channels for carrying data between tiles in different slices, each communication channel extending horizontally in an east-west direction. Each communication channel may be connected to each slice 2104 of the processor 2100.

[0236] The slices 2104 of processor 2100 may each correspond to different functions and may include arithmetic logic slices (e.g., FP / INT 2118), channel switching slices (e.g., NET 2120), and memory slices (e.g., MEM 2122). Arithmetic logic units can perform one or more arithmetic and / or logical operations on data received via communication channels to generate output data. Examples of arithmetic logic units may be matrix multiplication units and vector multiplication units. Memory slices include memory cells that store data. Memory slices can provide data to other slices via communication channels. Memory slices can also receive data from other slices via communication channels. Channel switching slices can configurably route data from one communication channel to any other communication channel. For example, data from a first channel can be provided to a second channel via a channel switching slice. In some embodiments, a channel switching slice can be implemented as a crossbar switch. Each slice 2104 also includes its own instruction queue (not shown) for storing instructions and an instruction control unit (ICU) for controlling instruction execution. Instructions in a given instruction queue can only be executed by a tile in its associated functional slice and not by other slices of the processor.

[0237] By arranging the tiles of processor 2100 into different functional slices 2104, the on-chip instruction and control flow of processor 2100 can be separated from the data flow. For example, according to some embodiments, Figure 21B One of the arrows illustrates the instruction flow within the processor architecture. According to at least one embodiment, Figure 21B Another arrow in the diagram illustrates the data flow within the processor architecture. As shown, instruction and control flow can flow across the tiles of processor 2100 in a first direction (e.g., north-south along the length of a functional slice, as indicated by the first arrow), while data flow can flow across the tiles of processor 2100 in a second direction (e.g., east-west across a functional slice, as indicated by the second arrow), which is perpendicular to the first direction.

[0238] Different functional slices of processor 2100 can correspond to MEM 2122 (memory), VXM (vector execution module), MXM (matrix execution module), NIM (numerical interpretation module), and SXM (swapping and permutation module). Each slice can include N tiles, all of which can be controlled by the same instruction control unit (ICU) (not shown). Each slice can operate completely independently and can only be coordinated using barrier-like synchronization primitives or by the compiler using tractable determinism. Each tile of processor 2100 can correspond to an execution unit organized as ×MSIMD tiles. For example, each tile of the on-chip memory of processor 2100 can be organized to atomically store L-element vectors. Therefore, MEM slices with N tiles can work together to store or process large vectors (e.g., with a total of N×M elements).

[0239] Tiles within a slice can execute instructions in an "interleaved" manner, where instructions can be issued tile-by-tile within the slice over N cycle periods. Functional slices can be physically arranged on the chip to allow for efficient data flow for pipelined execution over hundreds of cycles for common patterns. The data flow can perform a single "u-turn" (direction change) corresponding to a single matrix operation before being written back to memory; in some embodiments, a particular data flow can change direction multiple times before writing the resulting data back to memory (due to multiple matrix and vector operations).

[0240] When using a processor with a function slice architecture (e.g., a TSP), the TSP compiler (not shown) generates an explicit plan of how the processor 2100 can execute programs (e.g., microprograms). The compiler can specify when each operation will be executed, which function slices will perform the work, and which STREAM registers will hold operands. The compiler can maintain a high-fidelity (cycle-accurate) model of the hardware state of the processor 2100 (e.g., the TSP) so that the microprogram can coordinate data flow.

[0241] Processor 2100 (e.g., TSP) can use a web-hosted compiler that takes a model (e.g., an ML model, such as but not limited to a TensorFlow model) as input and issues a proprietary stream of instructions for processor 2100 (e.g., TSP). The compiler is responsible for coordinating the control and data flow of the program and specifying any instruction-level parallelism by explicitly bundling instructions that can and should be executed concurrently so that they can be dispatched together. The main hardware architecture includes the architecture-visible streaming register file (STREAM), which will be described in more detail below, and acts as a conduit for operands to flow from MEM slices (e.g., SRAM) to functional slices (and vice versa).

[0242] The MEM 2122 of processor 2100 can be used as: (1) a storage for model parameters, microprograms, and data on which they operate; and (2) an on-chip network (NoC) for transferring data operands from the MEM to functional slices and returning computation results to the MEM. In some embodiments, the on-chip memory may consume approximately 75% of the chip area of ​​processor 2100. In some embodiments, the on-chip memory of the MEM tile may include SRAM instead of DRAM due to the bandwidth requirements of processor 2100. The on-chip memory capacity of processor 2100 may be determined by: (i) the number of ML models that can reside on the chip simultaneously, (ii) the size of any given model, and (iii) partitioning of large models for adaptation to a multi-chip system. In some embodiments, the MEM system of processor 2100 may provide multiple memory slices organized into two distinct hemispheres (referred to as “MEM WEST” and “MEM EAST”, respectively).

[0243] The memory slices in each hemisphere can be mirrored such that the slices are physically numbered {0,...L} in the eastern hemisphere and {L,...0} in the western hemisphere, such that memory slice 0 in each hemisphere corresponds to the slice of the V×M slice closest to the hemisphere, where each hemisphere comprises L slices. The direction of data transfer towards the chip center can be referred to as inward, while data transfer towards the outer edge of the chip (easternmost or westernmost) can be referred to as outward. Although the memory hemispheres of processor 2100 can be referred to as east and west, it is understood that other names may be used to refer to different memory hemispheres in other embodiments.

[0244] In some embodiments, streaming register files (referred to as STREAMS) transfer operands and results between the SRAM of a MEM slice of processor 2100 and the functional slice. In some embodiments, multiple MEM slices (e.g., 2 to 10 adjacent MEM slices) can be physically organized into sets. Each slice set can be located between a pair of STREAMS register files, allowing each slice to read from or write to the STREAMS registers in either direction. By placing STREAMS register files between sets of MEM slices, the number of cycles required to transfer data operands across hemispheres can be reduced (e.g., reduced by a factor corresponding to the number of slices per set). The number of slices per set can be configured based on the distance of data transfer within a single clock cycle.

[0245] for Figure 21AAny processor and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. The API (and / or compiled instructions including the API) may be stored in any storage device internal or external to processor 2100 (e.g., in cache and / or memory). The results of the API may then be stored in storage devices internal or external to processor 2100, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0246] In at least one embodiment, processor 2100 may include one or more circuitry configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuitry, wherein one or more first instructions are scheduled in program order before one or more pause instructions; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configurable to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0247] Software Structure

[0248] The following figures illustrate, in a non-limiting manner, examples of software structures for implementing at least one embodiment.

[0249] Figure 22 A software stack of a programming platform according to at least one embodiment is illustrated. The programming platform may include a platform for accelerating computational tasks by utilizing hardware on a computing system. In at least one embodiment, software developers can access the programming platform through libraries, compiler instructions, and / or extensions to programming languages. The programming platform may be CUDA, Radeon Open Computing Platform (“ROCm”), OpenCL (OpenCL... TM Developed by the Khronos Group), SYCL, or Intel OneAPI.

[0250] The software stack 2200 of the programming platform can provide an execution environment for the application 2201. The application 2201 can include any computer software that can be launched on the software stack 2200. The application 2201 can include artificial intelligence (“AI”) / machine learning (“ML”) applications, high-performance computing (“HPC”) applications, virtual desktop infrastructure (“VDI”), or data center workloads.

[0251] Application 2201 and software stack 2200 run on hardware 2208. Hardware 2208 may include one or more GPUs, CPUs, FPGAs, AI engines, and / or other types of computing devices supporting programming platforms. Software stack 2200 may be vendor-specific and compatible only with vendor-specific devices, such as CUDA, ROCm, OneAPI, OpenCL, or other implementations. Hardware 2208 may include a host connected to one or more devices that can be accessed via application programming interface (“API”) calls to perform computational tasks. In at least one embodiment, the devices within hardware 2208 may include GPUs, FPGAs, AI engines, or other computing devices (but may also include CPUs) and their memory, while the host within hardware 2208 may include CPUs (but may also include computing devices) and their memory. For any hardware 2208 described above or elsewhere herein, the one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic, decoded by a processor decoder, scheduled (e.g., sequentially or out of order) for execution by a scheduler, executed by execution logic, reordered, and then retired by retirement logic. The API (and / or compiled instructions including the API) may be stored in any storage device (e.g., cache and / or memory) internal or external to hardware 2208. The results of the API may be stored in storage devices internal or external to hardware 2208, including registers, DRAM, flash memory, SRAM, cache, or other memory. One or more APIs described herein may receive calls. One or more APIs described herein may communicate with a library or a portion of a library to perform the function described by the call. One or more APIs described herein may receive calls and communicate with a library or a portion of a library to perform the function described by the call.

[0252] The software stack 2200 of the programming platform may include multiple libraries 2203, a runtime 2205, optional drivers / interfaces 2207, and device kernel drivers 2208. Each library 2203 may include data and programming code that can be used by computer programs and utilized during software development. Library 2203 may include pre-written code and subroutines, classes, values, type specifications, configuration data, documentation, help data, and / or message templates. Library 2203 may include functionality optimized for execution on one or more types of devices. Library 2203 may include functionality for performing mathematical, deep learning, and / or other types of operations on the device. Library 2203 may be associated with corresponding APIs 2202, which may include one or more APIs for exposing the functionality implemented in library 2203. A processor (e.g., CPU, GPU) may execute, call, or otherwise use one or more APIs to determine kernel priority. For example, a first kernel (e.g., a parent kernel) may launch a second kernel (e.g., a child kernel), and the processor may use the second kernel to launch an additional kernel (e.g., a grandchild kernel) independent of the first kernel. The processor can execute APIs or call APIs from memory to support dynamic stream priorities (e.g., updating priorities when performing operations using streams). For example, when the processor executes the API, it allows the programmer to copy stream priorities from one stream to one or more other streams.

[0253] Software stack 2200 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to set the stream's priority at any time after the stream is created. Software stack 2200 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to obtain the stream's current priority, where the priority is one of several attributes of the stream. Software stack 2200 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to obtain the stream's current priority as a single attribute. Software stack 2200 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to start the kernel to perform operations on the stream at a set priority, which may be different from the stream priority. Software stack 2200 may include an API for indicating whether an object (e.g., a thread synchronization object, such as, but not limited to, a barrier) tracks whether all data movement operations of a set of threads running on the GPU have a specified state after a specified time period, wherein the specified state may be a state indicating that data has been moved and is ready for use, and is specified using expected parity values ​​as input to the API.

[0254] Software stack 2200 may include one or more APIs for updating the kernel. The processor may execute APIs or call APIs from memory to update existing APIs, thereby supporting a context-independent kernel. This allows programmers to add kernel nodes to a graph without a graph context, so that the graph context can be dynamically associated with the kernel at runtime. Software stack 2200 may include one or more APIs that allow programmers to obtain kernel identifiers and graph contexts as separate parameters from kernel nodes, thereby enabling parameter retrieval from both the kernel and the context-independent kernel. Software stack 2200 may include one or more APIs that use parallel processors (e.g., but not limited to one or more graphics processing units) to initiate task graphs (e.g., task graphs) and execute one or more task graphs (e.g., including one or more programs).

[0255] Software stack 2200 may include one or more APIs for associating one or more instructions with one or more memory sorting operations (e.g., but not limited to fence or memory barrier operations). Instructions may be associated with one or more domains, causing memory sorting operations to execute in association with one or more specific domains without interfering with instructions in other domains. APIs may indicate that a thread has reached (e.g., reached a thread synchronization barrier) or completed a certain phase of work associated with an asynchronous data movement operation on the GPU. Software stack 2200 may include one or more APIs that allow programmers to manually indicate an expected transaction count when a thread completes a certain phase of work; this transaction count can be used to update an object used to track whether all data movement operations for a set of threads have been completed.

[0256] Application 2201 can be written as source code and then compiled into executable code, as described below. Figure 23 and Figure 24 As discussed in more detail, the executable code of application 2201 can run, at least partially, within the execution environment provided by software stack 2200. During the execution of application 2201, code that needs to run on the device (rather than the host) may be encountered. In this case, runtime 2205 can be invoked to load and launch the required code on the device. Runtime 2205 can include any technically feasible runtime system capable of supporting the execution of application 2201.

[0257] Runtime 2205 can be implemented as one or more runtime libraries associated with a corresponding API (shown as API 2204). One or more such runtime libraries may include functions for memory management, execution control, device management, error handling, and / or synchronization, etc. Memory management functions may include functions for allocating, dealing with, and copying device memory, as well as functions for transferring data between host memory and device memory. Execution control functions may include functions for launching functions on the device (sometimes referred to as the "kernel" when the function is a global function that can be called from the host) and setting attribute values ​​in buffers maintained by the runtime libraries so that a given function can be executed on the device.

[0258] Runtime libraries and corresponding APIs 2204 can be implemented in any technically feasible manner. One (or any number) of APIs can expose a set of low-level functions for fine-grained control of the device, while another (or any number) of APIs can expose a set of high-level functions for such functions. High-level runtime APIs can be built on top of low-level APIs. One or more runtime APIs can be language-specific APIs, which can be layered on top of language-independent runtime APIs.

[0259] Optional drivers or interfaces 2207 can be implemented, for example, for CUDA and ROCm implementations, which will be described further below. Optional drivers / interfaces 2207 can be associated with optional driver or interface APIs, such as, but not limited to, CUDA and / or ROCm APIs.

[0260] One or more processors disclosed in the “processing system” may execute, access, or otherwise use the software stack 2200. For example, the system-on-a-chip 900, parallel processor 1000, graphics multiprocessor 1034, processor 1100, processor 1200, accelerator 1300, neuromorphic processor 1405, supercomputer 1500, acceleration processing unit 1600, processor 1700, processor 1800, tensor processing unit 1900, processor 2000, and language processing unit 2100 may execute, use, call, or otherwise implement (e.g., by accessing memory) one or more APIs included in the software stack 2200.

[0261] Device kernel driver 2208 can be configured to facilitate communication with the underlying device. Device kernel driver 2208 can provide low-level functionality for APIs (such as, but not limited to, API 2204) and / or other software. Device kernel driver 2208 can be configured to compile intermediate representation (“IR”) code into binary code at runtime. For CUDA or other implementations (such as, but not limited to, ROCm, OneAPI, or OpenCL), device kernel driver 2208 can compile non-hardware-specific parallel thread execution (“PTX”) IR code into binary code for a specific target device at runtime (and cache the compiled binary code), which is sometimes referred to as “finalized” code. Doing so allows the finalized code to run on a target device that may not have existed when the source code was initially compiled into PTX code. Alternatively, the device source code can be compiled into binary code offline without device kernel driver 2208 compiling the IR code at runtime.

[0262] Processors described elsewhere in this document (e.g., but not limited to) Figure 9-21BThe processor (in the context) may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2200) to execute the tensor instructions concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous; or to execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; to add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; to indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or to execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0263] According to at least one embodiment, Figure 22 The software stack 2200 can execute in a CUDA implementation. The CUDA software stack 2200, on which the application 2201 can be launched, may include CUDA libraries 2203, CUDA runtime 2205, CUDA driver 2207, and device kernel driver 2208. The CUDA software stack 2200 can execute on hardware (e.g., a graphics multiprocessor 1134, which may include a CUDA-enabled GPU developed by NVIDIA Corporation, Santa Clara, California).

[0264] Application 2201, CUDA runtime 2205, and device kernel driver 2208 can perform the functions described above and elsewhere in this document. CUDA driver 2207 may include a library (libcuda.so) that implements CUDA driver API 2206. Similar to CUDA runtime API 2204 implemented by the CUDA runtime library (cudart), CUDA driver API 2206 exposes functions for memory management, execution control, device management, error handling, synchronization, and / or graphics interoperability. CUDA driver API 2206 differs from CUDA runtime API 2204 in that CUDA runtime API 2204 simplifies device code management by providing implicit initialization, context (similar to processes), and module (similar to dynamically loaded libraries) management. Compared to the high-level CUDA runtime API 2204, CUDA driver API 2206 can serve as a low-level API, providing finer-grained device control, especially in terms of context and module loading. CUDA Driver API 2206 can expose context management functions not exposed in CUDA Runtime API 2204. CUDA Driver API 2206 can also be language-agnostic, supporting technologies such as OpenCL in addition to CUDA Runtime API 2204. Furthermore, development libraries, including CUDA Runtime 2205, can be considered separate from driver components, including user-mode CUDA Driver 2207 and kernel-mode device driver 2208 (sometimes referred to as the "display" driver).

[0265] CUDA library 2203 may include mathematical libraries, deep learning libraries, parallel algorithm libraries, and / or signal / image / video processing libraries, which parallel computing applications (such as, but not limited to, application 2201) may utilize. CUDA library 2203 may include mathematical libraries, such as, but not limited to, the cuBLAS library (an implementation of the basic linear algebra subroutine (“BLAS”) for performing linear algebra operations), the cuFFT library (for computing the Fast Fourier Transform (“FFT”)), and the cuRAND library (for generating random numbers), etc. CUDA library 2203 may include deep learning libraries, such as, but not limited to, the cuDNN primitive library for deep neural networks and the TensorRT platform for high-performance deep learning inference, etc.

[0266] In at least one embodiment, the processor described elsewhere herein (e.g., but not limited to Figure) Figure 9-21BThe processor (in the context) may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2200) to execute the tensor instructions concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous; or to execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; to add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; to indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or to execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0267] According to at least one embodiment, Figure 22 The software stack 2200 can execute in the ROCm implementation. An application 2201 can be launched on the ROCm software stack 2200, which includes a language runtime 2203, a system runtime 2205, a thunk 2207, and a ROCm kernel driver 2208. The ROCm software stack 2200 executes on hardware 2209, which may include a ROCm-enabled GPU developed by AMD Inc. of Santa Clara, California.

[0268] Application 2201 is executable and can be combined with the above. Figure 22 Similar functionality to that discussed. Furthermore, the language runtime 2203 and system runtime 2205 can execute functions related to the above. Figure 22The runtime discussed is similar in functionality to 2205. The difference between the language runtime 2203 and the system runtime 2205 is that the system runtime 2205 is a language-independent runtime that implements the ROCr System Runtime API 2204 and uses the Heterogeneous System Architecture (“HSA”) runtime API. The HSA runtime API may include a streamlined user-mode API that exposes interfaces for accessing and interacting with the AMD GPU, including functions for memory management, execution control via kernel architecture dispatch, error handling, system and agent information, and runtime initialization and shutdown. Unlike the system runtime 2205, the language runtime 2203 may be an implementation of the language-specific runtime API 2202, which sits in a layer above the ROCr System Runtime API 2204. The language runtime API may include the Heterogeneous Computing Portable Interface (“HIP”) language runtime API, the Heterogeneous Computing Compiler (“HCC”) language runtime API, or the OpenCL API, etc. HIP is an extension of the C++ programming language, offering a functionally similar version to the CUDA mechanism. Furthermore, the HIP runtime API can include features related to the above. Figure 22 The discussion covers functions similar to those in the CUDA runtime API, such as, but not limited to, memory management, execution control, device management, error handling, and synchronization.

[0269] The Thunk(ROCt)2207 can be interface 2206, which is used to interact with the underlying ROCm driver 2208. The ROCm driver 2208 can be the ROCk driver, a combination of the AMD GPU driver and the HSA core driver (amdkfd). The AMD GPU driver can be a device core driver developed by AMD for GPUs, performing functions similar to those described above. Figure 22 The device kernel driver discussed is similar to the 2209. An HSA kernel driver can be a driver that allows different types of processors to share system resources more efficiently through hardware features.

[0270] Various libraries (not shown) may be included in the ROCm software stack 2200 on top of the language runtime 2203, and provide integration with the above. Figure 22 The discussion focuses on CUDA library 2203 and similar functionality. Various libraries can include mathematical libraries, deep learning libraries, and / or other libraries, such as, but not limited to, the hipBLAS library which implements functionality similar to CUDA cuBLAS, the rocFFT library for computing FFTs similar to CUDA cuFFT, etc.

[0271] Processors described elsewhere in this document (e.g., but not limited to) Figure 9-21BThe processor (in the context) may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2200) to execute the tensor instructions concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous; or to execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; to add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; to indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or to execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0272] According to at least one embodiment, Figure 22 The software stack 2200 can execute in an OpenCL implementation. The OpenCL software stack 2200, on which the application 2201 can be launched, may include the OpenCL framework 2203, the OpenCL runtime 2205, and a driver 2208. The OpenCL software stack 2200 can execute on non-vendor-specific hardware 2209. Because devices developed by different vendors support OpenCL, specific OpenCL drivers may be required for interoperability with the hardware of those vendors.

[0273] Application 2201, OpenCL runtime 2205, device kernel driver 2208, and hardware 2209 can execute in combination with the above. Figure 22Other implementations of the discussed application 2201, runtime 2205, device kernel driver 2208, and hardware 2209 offer similar functionality. Application 2201 may also include an OpenCL kernel (not shown), whose code will execute on the device.

[0274] OpenCL can define a "platform" that allows a host to control devices connected to it. The OpenCL framework provides platform-level APIs and runtime APIs, shown as Platform API 2202 and Runtime API 2204, respectively. Runtime API 2204 uses contexts to manage kernel execution on devices. Each identified device can be associated with a corresponding context, which Runtime API 2204 uses to manage the device's command queue, program objects, kernel objects, shared memory objects, and so on. Platform API 2202 exposes functions that allow the use of device contexts to select and initialize devices, submit work to devices via command queues, and enable data transfer with devices. In addition, the OpenCL framework provides various built-in functions (not shown), including mathematical functions, relational functions, and image processing functions.

[0275] The OpenCL framework 2203 may also include a compiler (not shown). Source code can be compiled offline before application execution or online during application execution. Unlike CUDA and ROCm, OpenCL applications can be compiled online by a compiler, which represents any number of compilers that can be used to compile source code and / or IR code (e.g., but not limited to Standard Portable Intermediate Representation (“SPIR-V”) code) into binary code. Alternatively, OpenCL applications can be compiled offline before execution.

[0276] In at least one embodiment, the processor described elsewhere herein (e.g., but not limited to) Figure 9-21BThe processor (in the context) may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2200) to execute the tensor instructions concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous; or to execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; to add one or more instructions to one or more store queues by a single thread in a cooperative thread array (CTA) of one or more SMs; to indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or to execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0277] According to at least one embodiment, the software may be supported by a programming platform configured to support various programming models, middleware and / or libraries, and frameworks that the application may rely on. The application may be an AI / ML application implemented using, for example, a deep learning framework (e.g., but not limited to, MXNet, PyTorch, or TensorFlow), which may rely on libraries such as, but not limited to, cuDNN, the NVIDIA Collective Communication Library (“NCCL”), and / or the NVIDIA Developer Data Loading Library (“DALI”) CUDA library to provide accelerated computation on the underlying hardware.

[0278] The programming platform can be a combination of the above. Figure 22The platform described is one of CUDA, ROCm, or OpenCL. The programming platform can support various programming models, which can be abstractions of the underlying computing system that allow the expression of algorithms and data structures. Programming models can expose features of the underlying hardware to improve performance. Programming models may include CUDA, HIP, OpenCL, C++ Accelerated Massive Parallelism (“C++AMP”), Open Multiprocessing (“OpenMP”), Open Accelerators (“OpenACC”), and / or Vulkan Compute.

[0279] Libraries and / or middleware can provide abstract implementations of programming models. Such libraries may include data and programming code that computer programs can use and leverage during software development. Such middleware may include software that provides services to applications beyond those offered by the programming platform. Libraries and / or middleware may include cuBLAS, cuFFT, cuRAND, and other CUDA libraries, or rocBLAS, rocFFT, rocRAND, and other ROCm libraries. Furthermore, libraries and / or middleware may include the NCCL and ROCm communication collection library (“RCCL”) libraries that provide communication routines for GPUs, the MIOpen library for accelerating deep learning, and / or the Eigen library for linear algebra, matrix and vector operations, geometric transformations, numerical solvers, and related algorithms.

[0280] Application frameworks may depend on libraries and / or middleware. Each application framework can be a software framework that provides a standard structure for implementing application software. Returning to the AI / ML example discussed above, AI / ML applications can be implemented using frameworks such as, but not limited to, deep learning frameworks like Caffe, Caffe2, TensorFlow, Keras, PyTorch, or MxNet.

[0281] In at least one embodiment, the processor described elsewhere herein (e.g., but not limited to) Figure 9-21BThe processor (in the context) may include one or more circuits configured to: execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous; or execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; add one or more instructions to one or more memory queues by a single thread in a cooperative thread array (CTA) of one or more SMs; indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or execute one or more instructions in an order independent of the order in which the instructions were submitted, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., the programming platform described herein) to execute the tensor instructions concurrently with one or more other instructions, at least in part, based on one or more indicators that at least one tensor instruction is asynchronous; or to execute one or more second instructions asynchronously in parallel with one or more first instructions to be executed by one or more first circuits, wherein one or more first instructions are scheduled before one or more pause instructions in program order; to add one or more instructions to one or more storage queues by a single thread in a cooperative thread array (CTA) of one or more SMs; to indicate to one or more threads that one or more matrix multiplication-accumulation (MMA) operations have been completed; or to execute one or more instructions or any of the operations described above or elsewhere herein by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator in an order independent of the order in which the instructions were submitted.

[0282] Figure 23 The method for using at least one embodiment in the above is shown. Figure 22Compiled code executed on one of the programming platforms shown. Compiler 2301 is configured to receive source code 2300, compile source code 2300, and output executable file 2310. Compiler 2301 can be configured to convert source code 2300 into host executable code 2307 for execution on a host and device executable code 2308 for execution on a device. Source code 2300 can be compiled offline before executing the application or can be compiled online during application execution. Source code 2300 can include code in any programming language supported by compiler 2301, such as, but not limited to, C++, C, Fortran, etc. Source code 2300 can be included in a single source file containing both host code and device code, indicating the location of the device code. The single source file can be a .cu file including CUDA code, or a .hip.cpp file including HIP code, or a file in other formats including both host code and device code. Alternatively, source code 2300 can include multiple source code files instead of a single source file, with host code and device code separated into these files. Compiler 2301 includes or can access one or more libraries to identify API call sequences to execute a single fused API, wherein the single fused API is a combination of two or more APIs. In at least one embodiment, compiler 2301 may be an NVIDIA CUDA compiler (“NVCC”) for compiling CUDA code in .cu files, or an HCC compiler for compiling HIP code in .hip.cpp files, or other compilers.

[0283] Compiler 2301 can be configured to compile source code 2300 into host executable code 2307 for execution on a host and device executable code 2308 for execution on a device. The operations performed by compiler 2301 include parsing source code 2300 into an abstract system tree (AST), performing optimizations, and generating executable code. When source code 2300 comprises a single source file, compiler 2301 can separate the device code from the host code in that single source file, compile the device code and host code into device executable code 2308 and host executable code 2307 respectively, and link the device executable code 2308 and host executable code 2307 together to form a single file.

[0284] Compiler 2301 may include compiler front-end 2302, host compiler 2305, device compiler 2306, and linker 2309. Compiler front-end 2302 may be configured to separate device code 2304 from host code 2303 in source code 2300. In at least one embodiment, device code 2304 may be compiled by device compiler 2306 into device executable code 2308, which may include binary code or IR code as described above. Separately, host code 2303 may be compiled by host compiler 2305 into host executable code 2307. For other compilers such as NVCC (e.g., but not limited to oneAPI, ROCm, and OpenCL compilers), host compiler 2305 may be a general-purpose C / C++ compiler that outputs native object code, while device compiler 2306 may be a low-level virtual machine (“LLVM”) based compiler that forks the LLVM compiler infrastructure and outputs PTX code or binary code. For HCC, both host compiler 2305 and device compiler 2306 may be LLVM based compilers that output object binary code.

[0285] After compiling source code 2300 into host executable code 2307 and device executable code 2308, linker 2309 can link host executable code 2307 and device executable code 2308 together to form executable file 2310. The host's native object code and the device's PTX or binary code can be linked together in an executable and linkable format (“ELF”) file, a container format for storing object code. Host executable code 2307 and device executable code 2308 can take any suitable format, such as, but not limited to, binary code and / or IR code. In at least one embodiment, for CUDA, host executable code 2307 may include native object code, while device executable code 2308 may include code in a PTX intermediate representation. In at least one embodiment, for ROCm, both host executable code 2307 and device executable code 2308 can include object binary code. Other implementations (e.g., but not limited to oneAPI, OpenCL) are considered and can be performed similarly to the CUDA and ROCm implementations described above.

[0286] Source code 2300 can be transformed before compilation. The source code is passed through a transformation tool (not shown), which transforms source code 2300 into transformed source code. Compiler 2301 can be used to compile the transformed source code into host executable code 2307 and device executable code 2308, a process similar to compiler 2301 compiling source code 2300 into host executable code 2307 and device executable code 2308, as described above. Figure 23The discussion.

[0287] The transformations performed by the transformation tools can be used to port Source Code 2300 to environments different from the environment it was originally intended to run in. The transformation tools may include a HIP converter, which is used to "hipify" CUDA code intended for the CUDA platform into HIP code that can be compiled and executed on the ROCm platform. The transformation of Source Code 2300 may include parsing Source Code 2300 and translating calls to APIs provided by one programming model (e.g., CUDA) into corresponding calls to APIs provided by another programming model (e.g., HIP), as described below. Figure 24 Let's discuss this in more detail. Returning to the example of HIP-enhanced CUDA code, calls to the CUDA runtime API, CUDA driver API, and / or CUDA libraries can be translated into corresponding HIP API calls. The automatic conversion performed by conversion tool 2301 may sometimes be incomplete, requiring additional manual intervention to fully port the source code 2300.

[0288] The techniques described herein can utilize various methods for converting one type of code into another. For example, compiler 2301 or other compilers described herein can convert a high-level language (e.g., source code as an abstraction of hardware) into a low-level language (e.g., machine code or intermediate representation). Source code can be scanned, parsed, transformed into an abstract syntax tree for semantic analysis, then converted into intermediate code, and finally into machine code or assembly language. Compiler 2301 or other compilers described herein may include a transpiler that can, for example, convert source code of one type into source code of another type, or convert one type of machine code into machine code of another type. Source code can be parsed and transformed into an abstract syntax tree, then into an intermediate model that can be transformed into an abstract syntax tree of the target language and generate code. Compiler 2301 or other compilers described herein can be used to achieve interoperability between different device architectures. For exam...

Claims

1. A processor, comprising: One or more circuits, the one or more circuits being used to execute the tensor instruction concurrently with one or more other instructions, based at least in part on one or more indicators that at least one tensor instruction is asynchronous.

2. The processor of claim 1, wherein the at least one tensor instruction and the one or more other instructions are added to one or more operation processing queues by a single thread in the cooperative thread array (CTA).

3. The processor of claim 1, wherein the at least one tensor instruction comprises a plurality of instructions to be divided into a first stage and a second stage, and the one or more circuits are further configured to execute one or more wait instructions, the one or more wait instructions causing the second stage to wait until an indication that the first stage has completed.

4. The processor of claim 1, wherein the at least one tensor instruction and the one or more other instructions are executed by at least one of the processing cores of the graphics processing unit (GPU) or the MMA accelerator, regardless of the order in which the tensor instruction or the one or more other instructions are submitted.

5. The processor of claim 1, wherein the at least one tensor instruction includes instructions for performing matrix multiplication-accumulation (MMA) operations.

6. The processor of claim 1, wherein one thread in the cooperative thread array is configured to wait for the tensor instruction to complete and for an indication that one or more operands of the tensor instruction are permitted to be accessed or modified.

7. The processor of claim 1, wherein the at least one tensor instruction comprises a plurality of instructions to be divided into a first stage and a second stage, and one or more instructions of the first stage are executed while the one or more other instructions modify the instructions of the second stage.

8. A system comprising: One or more processors, the one or more processors including one or more circuits, the one or more circuits being configured to execute the tensor instruction concurrently with one or more other instructions, at least in part based on one or more indicators that at least one tensor instruction is asynchronous.

9. The system of claim 8, wherein the at least one tensor instruction and the one or more other instructions are added to one or more operation processing queues by a single thread in the cooperative thread array (CTA).

10. The system of claim 8, wherein the at least one tensor instruction comprises a plurality of instructions to be divided into a first stage and a second stage, and the one or more circuits are further configured to execute one or more wait instructions that cause the second stage to wait until an indication that the first stage is complete.

11. The system of claim 8, wherein the at least one tensor instruction and the one or more other instructions are executed by at least one of the processing cores of a graphics processing unit (GPU) or an MMA accelerator, regardless of the order in which the tensor instruction or the one or more other instructions are submitted.

12. The system of claim 8, wherein the at least one tensor instruction includes instructions for performing matrix multiplication-accumulation (MMA) operations.

13. The system of claim 8, wherein one thread in the cooperative thread array is configured to wait for the tensor instruction to complete and for an indication that one or more operands of the tensor instruction are permitted to be accessed or modified.

14. The system of claim 8, wherein the at least one tensor instruction comprises a plurality of instructions to be divided into a first phase and a second phase, and one or more instructions of the first phase are executed while the one or more other instructions modify the instructions of the second phase.

15. A method comprising: The tensor instructions are executed concurrently with one or more other instructions, based at least in part on the fact that at least one tensor instruction is an asynchronous indicator.

16. The method of claim 15, wherein the at least one tensor instruction and the one or more other instructions are added to one or more operation processing queues by a single thread in the cooperative thread array (CTA).

17. The method of claim 15, further comprising: Indicate to the waiting thread that one or more matrix multiplication-accumulation operations caused by the at least one tensor instruction have been completed.

18. The method of claim 15, further comprising: The at least one tensor instruction and the one or more other instructions are executed, regardless of the order in which the at least one tensor instruction or the one or more other instructions are submitted.

19. The method of claim 15, further comprising: Waiting for an indication that at least a portion of the at least one tensor instruction has been completed; as well as Indicates that one or more operands of one or more first instructions are allowed to be accessed or modified.

20. The method of claim 15, further comprising: The at least one tensor instruction is divided into a first stage and a second stage; as well as The first stage is executed, while the second stage is modified in response to one or more other instructions.

Images