Application programming interface for pausing instructions

Abstract

This disclosure relates to application programming interfaces (APIs) for pausing instructions, and more specifically, to apparatus, systems, and techniques for executing one or more APIs to cause a graphics processing unit (GPU) to pause one or more instructions and / or to resume one or more instructions. In at least one embodiment, one or more processors, including one or more circuits, are used to execute the API to pause the execution of one or more GPU instructions, wherein the GPU instructions are associated with a context or application instructed to the API.

Description

[0001] Cross-references to related applications For all purposes, this application incorporates by reference the full disclosure of co-pending U.S. Patent Application No. 19 / 175,758, entitled “Application programming interface to RESUME INSTRUCTIONS,” which was filed concurrently. Technical Field

[0002] At least one embodiment relates to a processor for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions. In at least one embodiment, the processor, including a circuit system, is used to execute the API to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API. Background Technology

[0003] Applications running on a GPU may include instructions that the user wants to execute in different ways, such as in a different order. Techniques for generating such instructions and for executing them in a non-initial order (e.g., in the order the user expects) can be improved. Attached Figure Description

[0004] Figure 1 This is a block diagram illustrating a system 100 for executing one or more APIs associated with pause and resume GPU instructions, according to at least one embodiment; Figure 2A and Figure 2B This is a block diagram illustrating a system for suspending instructions associated with one or more contexts according to at least one embodiment; Figure 3A and Figure 3B This is a block diagram illustrating a system for recovering instructions associated with one or more contexts according to at least one embodiment; Figure 4 This is a block diagram illustrating a system for using an API to suspend context-associated instructions of a processor, according to at least one embodiment. Figure 5 This is a process flow diagram illustrating an API for suspending context-associated instructions of a processor according to at least one embodiment; Figure 6 This is a block diagram illustrating a system for using an API to suspend instructions associated with an application, according to at least one embodiment. Figure 7This is a process flow diagram illustrating an API for suspending instructions associated with an application, according to at least one embodiment. Figure 8 This is a block diagram illustrating a system for using an API to enable a processor to resume execution of context-associated instructions, according to at least one embodiment. Figure 9 This is a process flow diagram illustrating an API for enabling a processor to resume execution of context-associated instructions, according to at least one embodiment. Figure 10 This is a block diagram illustrating a system for using an API to enable a processor to resume execution of instructions associated with an application, according to at least one embodiment. Figure 11 This is a flowchart illustrating an API for resuming the execution of instructions associated with an application, according to at least one embodiment. Figure 12 The diagram illustrates a processor and a storage device according to at least one embodiment; Figure 13 An example data center system according to at least one embodiment is shown; Figure 14 A system-on-a-chip (SOC) according to at least one embodiment is shown. Figure 15A A parallel processor according to at least one embodiment is shown; Figure 15B A processing cluster according to at least one embodiment is shown; Figure 15C A graphics multiprocessor according to at least one embodiment is shown; Figure 16 An accelerator processor according to at least one embodiment is shown; Figure 17A A central processing unit according to at least one embodiment is shown; Figure 17B The illustration shows an embodiment according to at least one of the embodiments. Figure 17A The core of the central processing unit; Figure 18 Another accelerator processor according to at least one embodiment is shown; Figure 19 A neuromorphic processor according to at least one embodiment is shown; Figure 20 A supercomputer according to at least one embodiment is shown; Figure 21 Another accelerator processor according to at least one embodiment is shown; Figure 22 Another processor according to at least one embodiment is shown; Figure 23 Another accelerator processor according to at least one embodiment is shown; Figure 24 A tensor processing unit according to at least one embodiment is shown; Figure 25 A RISC-V compatible processor according to at least one embodiment is shown; Figure 26A and Figure 26B A language processing unit according to at least one embodiment is shown; Figure 27 A software stack of a programming platform according to at least one embodiment is shown; Figure 28 Software supported by a programming platform according to at least one embodiment is shown; Figure 29 A method for using at least one embodiment is shown. Figure 28 Compiled code executed on the programming platform; Figure 30 An example of an autonomous vehicle and its system architecture according to at least one embodiment is shown; Figure 31A The inference and / or training logic according to at least one embodiment is illustrated; Figure 31B The inference and / or training logic according to at least one embodiment is shown; and Figure 31C Training and deployment of a neural network according to at least one embodiment are illustrated. Detailed Implementation

[0005] Numerous specific details are set forth in the following description to provide a more thorough understanding of at least one embodiment. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0006] In an exemplary embodiment, the GPU executes instructions in the order they are queued by the host program. In some cases, technical challenges arise if higher-priority instructions should be executed before other instructions already in the queue. For example, a programmer might want to write code that sends additional instructions to the GPU and executes them before queued instructions. In this exemplary embodiment, the processor includes circuitry for executing APIs to pause or resume the execution of one or more GPU instructions. In some cases, these instructions may be associated with a GPU context. In other cases, they may be associated with an application. Furthermore, the application may have multiple contexts, so pausing instructions associated with the application also pauses instructions associated with each of these multiple contexts.

[0007] In another aspect of this example, an API for pausing the execution of GPU instructions causes the instructions indicated by using that API to be removed from or deactivated from the instruction queue on the GPU. In embodiments of this example, these instructions may correspond to GPU kernels. In one embodiment, the API for pausing the execution of GPU instructions can cause the scheduler to stop scheduling the indicated instructions for execution on the GPU. In some embodiments, the API for pausing the execution of GPU instructions can restore the state of the thread executing the GPU instructions.

[0008] In another aspect of this example, the API for resuming the execution of GPU instructions can re-add or reactivate the instruction whose execution was scheduled to be performed on the GPU by using the API. In one embodiment, the API for resuming the execution of GPU instructions can cause the scheduler to resume scheduling the indicated instruction for execution on the GPU. In some embodiments, the API for resuming the execution of GPU instructions can restore the state of a thread that was executing GPU instructions before being paused.

[0009] In another aspect of this example, these APIs can be used to control the relative execution priority between different contexts within an application and between different applications. For example, within an application, an API for pausing GPU instructions associated with a context can be used to give higher execution priority to other contexts within the same application. Similarly, an API for pausing GPU instructions associated with a host program can be used to give higher execution priority to other applications. In at least one embodiment, these APIs can be used to enable time slice sharing between instructions associated with a first context or host program and instructions associated with a second context or host program.

[0010] Figure 1 This is a block diagram illustrating a system 100 for executing one or more application programming interfaces (APIs) associated with pausing and resuming GPU instructions, according to at least one embodiment. In at least one embodiment, system 100 includes a software program 102, a driver / runtime 104, one or more libraries 106, a GPU 114, and / or one or more APIs 110.

[0011] Technical issues may include: GPU 114 executes instructions (e.g., kernels) in the order the host program queues them, even if higher-priority instructions need to be executed before other instructions already in the queue. Users (e.g., programmers or other engineers) may want to be able to use an API to send instructions to GPU 114 for execution so that these instructions are executed before queued instructions.

[0012] The embodiments described herein are with reference to GPU 114, which may include various dedicated circuitry for performing workloads associated with, for example, graphics rendering, video playback, artificial intelligence, and simulation. Examples of GPUs may include those related to… Figure 12 -31 Related descriptions or descriptions of GPUs elsewhere in this document. Furthermore, while the term GPU is used herein with respect to various embodiments, similar or corresponding embodiments may be practiced using other types of parallel processors and accelerators, including those described above. Figure 12 The processors and accelerators described in -31 or elsewhere in this document.

[0013] In at least one embodiment, system 100 includes API 110 (e.g., cuCtxPauseExecution()) for pausing 116 one or more GPU instructions associated with a context, for example by indicating a first instruction to execute a second instruction before the first instruction. The input to cuCtxPauseExecution() API 110 may include an identifier of the GPU context. When executed, the API causes the GPU to pause execution of all instructions currently scheduled for execution for the indicated GPU context. To this end, API 110 may cause the GPU scheduler to stop scheduling new instructions associated with that context, wait for pending instructions associated with that context to complete, and then store state information, such as the state of registers and the program counter. API 110 returns a success or failure indicator.

[0014] In at least one embodiment, system 100 includes an API 110 (e.g., cuCtxPauseAll()) for pausing 118 one or more instructions associated with an application, such as pausing the execution of all one or more cores of the GPU currently executing for the application. The application may have multiple different GPU contexts, so API 110 can pause instructions associated with multiple different contexts. When executed, the API can cause the GPU to pause execution of all instructions currently scheduled for execution by that GPU for the application (also referred to as the host program or other related terms). To this end, the API can cause the GPU scheduler to stop scheduling new instructions for the host program, wait for pending instructions associated with that host to complete, and then store state information, such as the state of registers and the program counter. The API can return a success or failure indicator.

[0015] In at least one embodiment, system 100 includes API 110 (e.g., cuCtxResumeExecution()) for resuming one or more instructions associated with context at 120. For example, the GPU can resume execution of one or more instructions at 120 that were previously paused (e.g., by API 110). The input to the cuCtxResumeExecution() API may include identifiers of the one or more instructions to be resumed, such as one or more instructions previously paused using cuCtxPauseExecution(). When executed, the API can use stored state information to restore the state of threads in the group and use a program counter stored for each thread to resume each thread starting from the instruction following the last completed instruction.

[0016] In at least one embodiment, system 100 includes API 110 (e.g., cuCtxResumeAll()) for resuming 122 one or more instructions associated with an application, such as all kernels associated with the application. In an embodiment, this API may not receive any parameters as input, but instead obtain information about the associated program by invoking other APIs to determine the identity of the calling application. When executed, this API enables the GPU to resume execution of all threads that were executed on behalf of the host program and subsequently suspended. To do this, the API may identify all suspended threads that the host program calling the API is to execute, restore their state (e.g., values ​​stored in the thread's registers), and begin execution of each thread starting with the instruction immediately following the last instruction executed before the suspension. The API may return a success or failure indicator.

[0017] In at least one embodiment, the software program 102 is a software module, such as... Figure 12 The software modules described herein. In at least one embodiment, the software program 102 includes one or more software modules. In at least one embodiment, the software modules are as follows: Figure 12 Described non-exclusively. In at least one embodiment, one or more APIs 110 are software instruction sets that, when executed, cause one or more processors to perform one or more computational operations. In at least one embodiment, one or more APIs 110 are distributed or otherwise provided as part of one or more libraries 106, runtimes 104, drivers 104, and / or any other grouping of software and / or executable code further described herein. In at least one embodiment, one or more APIs 110 perform one or more computational operations in response to a call to software program 102. In at least one embodiment, software program 102 is a collection of software code, commands, instructions, or other text sequences for instructing a computing device to perform one or more computational operations and / or call one or more other instruction sets (e.g., API 110 or API function 112) to be executed. In at least one embodiment, the functionality provided by one or more APIs 110 includes software functions 112, such as software functions that can be used to accelerate one or more portions of software program 102 using one or more parallel processing units (PPUs) (e.g., graphics processing units (GPUs)). In at least one embodiment, the software program is a neural network. In at least one embodiment, the software program includes one or more algorithms.

[0018] In at least one embodiment, API 110 is a hardware interface to one or more circuits for performing one or more computational operations. In at least one embodiment, the one or more software APIs 110 described herein are implemented as one or more circuits for performing combined... Figure 1-12 One or more of the aforementioned technologies. In at least one embodiment, one or more software programs 102 include instructions that, if executed, cause one or more hardware devices and / or circuits to perform a combination. Figure 1-12 One or more of the aforementioned technologies.

[0019] In at least one embodiment, software program 102 (e.g., a user-implemented software program) utilizes one or more application programming interfaces (APIs) 110 to perform various computational operations, such as memory reservation, matrix multiplication, arithmetic operations, or any computational operation performed by a parallel processing unit (PPU) (e.g., a graphics processing unit (GPU)), as further described herein. In at least one embodiment, one or more APIs 110 provide a set of callable functions 112 (referred to herein as APIs, API functions, and / or functions), which each perform one or more computational operations, such as computational operations related to parallel computing. In at least one embodiment, one or more APIs 110 provide functions 112 for pausing 116 one or more instructions associated with a context (e.g., the kernel), pausing 118 one or more instructions associated with an application, resuming 120 one or more instructions associated with a context, resuming 122 one or more instructions associated with an application, and / or otherwise performing the operations described herein. In at least one embodiment, one or more APIs 110 provide functions 112 for pausing 118 one or more instructions associated with the application, resuming 120 one or more instructions associated with the context, resuming 122 one or more instructions associated with the application, and / or otherwise performing the operations described herein.

[0020] In at least one embodiment, one or more software programs 102 interact with or otherwise communicate with one or more APIs 110 to perform one or more computational operations using one or more PPUs (e.g., GPU 114). In at least one embodiment, the one or more computational operations using one or more PPUs include at least one or more sets of computational operations that are accelerated by being performed at least partially by said one or more PPUs. In at least one embodiment, one or more software programs 102 interact with one or more APIs 110 to facilitate parallel computing using remote or local interfaces.

[0021] In at least one embodiment, the interface is software instructions that, when executed, provide access to one or more functions 112 provided by one or more APIs 110. In at least one embodiment, when a software developer compiles one or more software programs 102 in conjunction with one or more libraries 106 that include one or more APIs 110 or otherwise provide access to one or more APIs 110, the software program 102 uses the native interface. In at least one embodiment, one or more software programs 102 are statically compiled in conjunction with precompiled libraries 106 or uncompiled source code including instructions for executing one or more APIs 110. In at least one embodiment, one or more software programs 102 are dynamically compiled, and the one or more software programs are linked to one or more precompiled libraries 106 that include one or more APIs 110 using a linker.

[0022] In at least one embodiment, when a software developer executes a software program, the software program 102 uses a remote interface that communicates with a library 106 including one or more APIs 110, or otherwise via a network or other remote communication medium. In at least one embodiment, the library 106 including one or more APIs 110 is executed by a remote computing service (e.g., a computing resource service provider). In another embodiment, the library 106 including one or more APIs 110 is executed by any other computing host that provides the one or more APIs 110 to the software program 102.

[0023] In at least one embodiment, one or more software programs 102 utilize one or more APIs 110 to allocate and otherwise manage memory to be used by the software program 102. In at least one embodiment, one or more software programs 102 utilize one or more APIs 110 to allocate and otherwise manage memory to be used by one or more portions of the software program 102, which will be accelerated using one or more PPUs (e.g., GPUs or any other accelerators or processors further described herein). In at least one embodiment, these software programs 102 use API 110 to execute one or more functions 112 and / or perform the operations described herein.

[0024] In at least one embodiment, API 110 is an API for facilitating parallel computing. In at least one embodiment, API 110 is any other API further described herein. In at least one embodiment, API 110 is provided by a driver and / or runtime 104. In at least one embodiment, API 110 is provided by a CUDA user-mode driver. In at least one embodiment, API 110 is provided by a CUDA runtime. In at least one embodiment, driver 104 is a data value and software instruction that, if executed, performs or otherwise facilitates the operation of one or more functions 112 of API 110 during the loading and execution of one or more portions of software program 102. In at least one embodiment, runtime 104 is a data value and software instruction that, if executed, performs or otherwise facilitates the operation of one or more functions 112 of API 110 during the execution of software program 102. In at least one embodiment, one or more software programs 102 utilize one or more APIs 110 implemented or otherwise provided by a driver and / or runtime 104 to pause 116 one or more instructions associated with a context, pause 118 one or more instructions associated with an application, resume 120 one or more instructions associated with a context, and / or resume 122 one or more instructions associated with an application; wherein the neural network is selected by one or more software programs 102 during execution by one or more PPUs (e.g., GPUs).

[0025] In at least one embodiment, one or more software programs 102 utilize one or more APIs 110 provided by the driver and / or runtime 104 to suspend 116 one or more instructions associated with a context, suspend 118 one or more instructions associated with an application, resume 120 one or more instructions associated with a context, and / or resume 122 one or more instructions associated with an application. In at least one embodiment, one or more APIs 110 provide software programs for suspending 116 one or more instructions associated with a context, suspending 118 one or more instructions associated with an application, resuming 120 one or more instructions associated with a context, and / or resuming 122 one or more instructions associated with an application, as described above, by sending instructions to the GPU via the driver and / or runtime 104. In at least one embodiment, one or more software programs 102 utilize one or more APIs 110 provided by the driver and / or runtime 104 to allocate or otherwise reserve one or more blocks of memory for one or more PPUs (e.g., GPU 114). In at least one embodiment, one or more software programs 102 utilize one or more APIs 110 provided by the driver and / or runtime 104 to indicate, allocate, or otherwise reserve memory blocks. In at least one embodiment, one or more APIs 110 are used to pause 118 one or more instructions associated with the application, resume 120 one or more instructions associated with the context, and / or resume 122 one or more instructions associated with the application, as in combination. Figure 1-12 As shown in any of the figures.

[0026] In at least one embodiment, to improve the usability of software program 102 and / or optimize one or more portions of software program 102 that will be accelerated by one or more PPUs (e.g., GPUs), in one embodiment, one or more APIs 110 provide one or more API functions 112 for pausing 116 one or more instructions associated with a context, pausing 118 one or more instructions associated with an application, resuming 120 one or more instructions associated with a context, and / or resuming 122 one or more instructions associated with an application, as described above and in combination. Figure 1-12 Further described. In at least one embodiment, exemplary system 100 includes processor 1008 (see...) Figure 6The processor includes one or more circuitry for executing one or more software programs to cause one or more functions 112. In at least one embodiment, an exemplary block diagram depicts a system 100 including one or more processors for executing one or more software programs to suspend 116 one or more instructions associated with a context, suspend 118 one or more instructions associated with an application, resume 120 one or more instructions associated with a context, and / or resume 122 one or more instructions associated with an application. In at least one embodiment, the processor uses API 110 to suspend 116 one or more instructions associated with a context, suspend 118 one or more instructions associated with an application, such that GPU 120 resumes execution of instructions, resumes 122 one or more instructions associated with an application, to otherwise perform the operations described herein, and / or combinations thereof.

[0027] In at least one embodiment, system 100 includes a collection of one or more hardware and / or software computing resources, including instructions that, when executed, implement one or more communication processes, such as those described herein. In at least one embodiment, system 100 is a software program, an application program, and / or variations thereof, executing on computer hardware. In at least one embodiment, one or more processes of system 100 are executed in any suitable manner by any suitable processing system or unit described below (e.g., graphics processing unit (GPU), general-purpose GPU (GPGPU), parallel processing unit (PPU), central processing unit (CPU), data processing unit (DPU)), including sequential execution, parallel execution, and / or variations thereof. In at least one embodiment, system 100 uses a machine learning training framework (e.g., PYTORCH, TENSORFLOW, BOOST, CAFFE, MICROSOFT COGNITIVE TOOLKIT / CNTK, MXNET, CHAINER, KERAS, DEEPLEARNING4J, and / or other training frameworks) to implement and perform the operations described herein for executing the API. In at least one embodiment, as an example, training a neural network model includes using a server (e.g., an NVIDIA DGX server) that also includes at least a GPU (e.g., an AMD MI200, VEGAL10, VEGO20, and ARCTURUS), an optimizer (e.g., an ADAM OPTIMIZER), or a discriminator architecture (e.g., a discriminator architecture from face-vid2vid for training with GAN loss).

[0028] In at least one embodiment, system 100 includes one or more modules that enable the system to execute one or more APIs described herein. In at least one embodiment, the modules include any combination of logic (e.g., software, hardware, firmware) and / or circuitry configured to perform the functions described herein. In at least one embodiment, the modules include one or more circuits that form part of a larger system (e.g., integrated circuit (IC), system-on-a-chip (SoC), central processing unit (CPU), graphics processing unit (GPU), data processing unit (DPU), etc.). In at least one embodiment, the controller includes any combination of logic (e.g., software, hardware, firmware) and / or circuitry configured to perform the functions described herein. In at least one embodiment, the software includes software packages, code, programming languages, drivers, instructions, instruction sets, or any combination thereof. In at least one embodiment, the hardware includes hardwired circuitry, programmable circuitry, state machine circuitry, fixed-function circuitry, execution unit circuitry, firmware storing instructions executed by the programmable circuitry, or any combination thereof.

[0029] In at least one embodiment, system 100 includes a logic unit. In at least one embodiment, the logic unit includes firmware logic, hardware logic, or any combination thereof, configured to provide any of the functions further described herein. In at least one embodiment, the logic unit includes a circuit system that forms part of a larger system (e.g., IC, SoC, CPU, GPU, DPU). In at least one embodiment, the logic unit includes a logical circuit system for implementing firmware and / or hardware to perform one or more APIs described herein.

[0030] In at least one embodiment, system 100 includes an engine. In at least one embodiment, the engine includes modules and / or logic units as further described herein. In at least one embodiment, the component includes modules and / or logic units as further described herein. In at least one embodiment, the engine includes software logic, firmware logic, hardware logic, or a combination thereof, configured to provide any functionality further described herein. In at least one embodiment, the component includes software logic, firmware logic, hardware logic, or a combination thereof, configured to provide any functionality further described herein. In at least one embodiment, operations performed by hardware and / or firmware may alternatively be implemented via software modules, which may be embodied as software packages, code, and / or instruction sets. In at least one embodiment, the logic unit may also utilize a portion of software to implement its functionality.

[0031] In at least one embodiment, system 100 includes one or more processors for executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 is as follows Figure 1-12 The system shown is included in, for example Figure 1-12 The system shown and / or otherwise includes, for example Figure 1-12 The system shown is configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), may be used to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0032] In at least one embodiment, system 100 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 executes... Figure 1-12 The illustrated one or more processes, for example, executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), may be used to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0033] In at least one embodiment, system 100 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 is Figure 1-12 The system shown is included Figure 1-12 The system shown or otherwise includes Figure 1-12 The system shown is configured to execute an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), are used to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0034] In at least one embodiment, system 100 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is configured to execute an application programming interface (API) to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 executes... Figure 1-12 The illustrated one or more processes, for example, executing an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein these GPU instructions are recognized by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 100 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), may be used to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0035] Figure 2A and Figure 2B This is a block diagram of a system 200 according to at least one embodiment, the system being used to suspend instructions associated with one or more GPU contexts. In at least one embodiment, system 200 includes one or more GPUs (e.g., GPU host 202), one or more engines 206, one or more contexts, and / or one or more time slice groups (TSGs). In at least one embodiment, one or more GPU hosts 202 execute one or more applications to execute one or more instructions (e.g., kernels) corresponding to one or more contexts. In at least one embodiment, system 200 includes one or more contexts (e.g., contexts A, B, and / or C) belonging to one or more time slice groups (TSGs) 204, such as TSG 1 204A, TSG 2 204B, and / or TSG 3 204C.

[0036] In at least one embodiment, system 200 executes one or more APIs 110 (e.g., cuCtxPauseExecution()). In at least one embodiment, system 200 uses API 110 that allows one or more users to pause one or all CUDA contexts within an application (e.g., context A of TSG 1 204A, context B of TSG 2 204B, and / or context C of TSG 3 304C), thereby enabling the immediate execution of high-priority instructions by preempting ongoing work.

[0037] In at least one embodiment, for a CUDA context supported by a dedicated time slice group (TSG), system 200 uses one or more synchronization APIs 110 to signal to the kernel-mode driver (KMD) to (i) disable the TSG and then preempt the TSG to pause and / or (ii) enable the TSG to resume the CUDA context. In at least one embodiment, a "disable" signal followed by a "preempt" signal can remove the TSG from the runlist and stop all running instructions within that TSG. In at least one embodiment, when the TSG is re-enabled via a "resume" signal, it is re-added to the runlist, thereby allowing the instructions in that TSG to resume execution.

[0038] In at least one embodiment, in the case of a shared TSG CUDA context (e.g., those managed by a Multi-Process Service (MPS)), one or more API 110 calls will stall work that has not yet been picked up by the ESCHED unit. In at least one embodiment, this is achieved by including a semaphore acquisition method at each startup. In at least one embodiment, the semaphore remains released during normal operation but is modified during paused API 110 calls to prevent the submission of more queued work.

[0039] In at least one embodiment, system 200 extends control to an external process by utilizing an interrupt handler thread in CUDA that listens for pause and resume signals via a file descriptor. In at least one embodiment, system 200 allows end users to manage one or more CUDA contexts within an application from outside the application process. In at least one embodiment, system 200 grants users and application administrators fine-grained control over GPU resources by using one or more APIs 110. In at least one embodiment, system 200 includes implementing custom scheduling logic to allow efficient management of CUDA contexts to meet real-time performance requirements.

[0040] In at least one embodiment, system 200 includes one or more engines 206. In at least one embodiment, engine 206 includes modules and / or logic units as further described herein. In at least one embodiment, components include modules and / or logic units as further described herein. In at least one embodiment, engine 206 includes software logic, firmware logic, hardware logic, or a combination thereof, configured to provide any functionality as further described herein. In at least one embodiment, components include software logic, firmware logic, hardware logic, or a combination thereof, configured to provide any functionality as further described herein. In at least one embodiment, operations performed by hardware and / or firmware may alternatively be implemented via software modules, which may be embodied as software packages, code, and / or instruction sets. In at least one embodiment, logic units may also utilize a portion of software to implement their functionality. In at least one embodiment, a context is an environment in which one or more kernels execute, such as state-related information (e.g., memory allocation and kernel configuration). In at least one embodiment, queues may also be illustrated in other ways by executing one or more instructions (e.g., kernels) according to one or more run lists of one or more contexts.

[0041] In at least one embodiment, one or more engines 206 send one or more signals to the GPU host 202 to initiate a pause of one or more contexts, such as context A in TSG 1 204A. In at least one embodiment, when context A is paused 210, the GPU continues to execute one or more next time slice groups 204, such as TSG 2 204B and TSG 3 204C.

[0042] In at least one embodiment, system 200 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 performs... Figure 1-12The processes shown herein include, for example, executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), are shown to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0043] In at least one embodiment, system 200 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0044] In at least one embodiment, system 200 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 200 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0045] Figure 3A and Figure 3B This is a block diagram illustrating a system 300 for recovering one or more contexts according to at least one embodiment. In at least one embodiment, system 300 includes one or more GPUs (e.g., GPU host 302), one or more engines 306, one or more contexts, and / or one or more time slice groups (TSGs). In at least one embodiment, one or more GPU hosts 302 execute one or more applications to execute one or more instructions (e.g., kernels) corresponding to one or more contexts. In at least one embodiment, system 300 includes one or more contexts (e.g., contexts A, B, and / or C) belonging to one or more time slice groups (TSGs) 304, such as TSG 1 304A, TSG 3 304B, and / or TSG 3 304C.

[0046] In at least one embodiment, system 300 includes one or more engines 306. In at least one embodiment, engine 306 includes modules and / or logic units as further described herein. In at least one embodiment, components include modules and / or logic units as further described herein. In at least one embodiment, engine 306 includes software logic, firmware logic, hardware logic, or a combination thereof, configured to provide any functionality as further described herein. In at least one embodiment, components include software logic, firmware logic, hardware logic, or a combination thereof, configured to provide any functionality as further described herein. In at least one embodiment, operations performed by hardware and / or firmware may alternatively be implemented via software modules, which may be embodied as software packages, code, and / or instruction sets. In at least one embodiment, logic units may also utilize a portion of software to implement their functionality. In at least one embodiment, a context is an environment in which one or more kernels execute, such as state-related information (e.g., memory allocation and kernel configuration). In at least one embodiment, a queue may be otherwise illustrated by executing one or more instructions (e.g., kernels) according to one or more run lists of one or more contexts.

[0047] In at least one embodiment, system 300 executes one or more APIs 110, such as instructions to restore 120 and / or 122 contexts associated with the application (e.g., corresponding to one or more contexts and one or more kernels of TSG 304). In at least one embodiment, one or more engines 306 send one or more signals to GPU host 302 to initiate the restoration of one or more previously suspended contexts, such as context A suspended 310 in TSG 1 304A. In at least one embodiment, when context A of suspended 310 is restored, host GPU 302 then returns context A to the run list in its time slice group 304, such as TSG 1 304A. In at least one embodiment, system 300 returns context A of suspended 310 to the run list after receiving a signal from API 110, for example, context A is restored 312 in TSG 1 304A.

[0048] In at least one embodiment, system 300 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 300 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is configured to execute an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 300 executes... Figure 1-12 The processes shown include, for example, executing an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 300 includes... Figure 13-31C The hardware shown herein includes one or more pieces of hardware, such as those used to execute an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0049] In at least one embodiment, system 300 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 300 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 300 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein these GPU instructions are recognized by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 300 includes Figure 13-31COne or more pieces of hardware, such as those used to execute an application programming interface (API), to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0050] Figure 4 This is a block diagram illustrating a system 400 for using an API to cause a processor to pause instructions, according to at least one embodiment. In at least one embodiment, system 400 includes one or more APIs 110 (e.g., for pausing one or more instructions 116 associated with context 404), one or more processors (e.g., host GPU 202, see below). Figure 2A-2B ), one or more inputs 402 (e.g., one or more contexts 404), one or more engines 202 and / or one or more applications 406, one or more instructions executed using one or more contexts 404 (e.g., application 406) and / or one or more components described herein.

[0051] In at least one embodiment, system 400 executes API 110 (e.g., cuCtxPauseExecution()) to pause one or more instructions associated with context 404 of 116, for example by indicating a first instruction that a second instruction will be executed before the first instruction. In at least one embodiment, one or more inputs 402 for executing API 110 to pause one or more instructions associated with context 404 of 116 will include an indication of one or more contexts 404 to be paused (e.g., an indication of the first instruction) and / or a second identifier of the second instruction to be executed before the first instruction. In at least one embodiment, the indication includes one or more identifiers, such as a kernel identifier. In at least one embodiment, when system 400 uses API 110 to pause one or more instructions associated with context 404 of 116, the API pauses one or more instructions of application 406 (e.g., kernel), and the processor executes the next instruction and / or places the second instruction at the beginning of the queue of instructions (e.g., kernel) to be executed by the GPU within one clock cycle. As an example, in at least one embodiment, if context 3 is paused in the TSG, the GPU will continue to execute the next instruction in context 4 of the subsequent time slice.

[0052] In at least one embodiment, system 400 executes process 500 to pause one or more instructions associated with context 404 using API 110. In at least one embodiment, the output 408 of executing API 110 to pause one or more instructions associated with context 404 will include an indication 410 indicating whether the GPU successfully paused the instructions. In at least one embodiment, in system 400, the GPU provides output 408 as a response to executing API 110, such as output 408 including one or more success indications 410 (e.g., flags), such as "success," "failure," and / or variations thereof.

[0053] In at least one embodiment, system 400 includes a collection of one or more hardware and / or software computing resources, including instructions that, when executed, can perform one or more communication processes, such as those described herein. In at least one embodiment, system 400 is a software program running on computer hardware, an application program running on computer hardware, and / or variations thereof. In at least one embodiment, one or more processes of system 400 are executed in any suitable manner by any suitable processing system or unit described below (e.g., graphics processing unit (GPU), general-purpose GPU (GPGPU), parallel processing unit (PPU), central processing unit (CPU)), data processing unit (DPU), including sequential execution, parallel execution, and / or variations thereof. In at least one embodiment, system 400 uses machine learning training frameworks (e.g., PYTORCH, TENSORFLOW, BOOST, CAFFE, MICROSOFT COGNITIVE TOOLKIT / CNTK, MXNET, CHAINER, KERAS, DEEPLEARNING4J, and / or other training frameworks) to implement and perform the operations described herein to select one or more neural networks to perform one or more wireless signal operations. In at least one embodiment, as an example, training a neural network model includes using a server (e.g., an NVIDIA DGX server) that also includes at least a GPU (e.g., an AMD MI200, VEGAL10, VEGO20, and ARCTURUS), an optimizer (e.g., an ADAMOPTIMIZER), or a discriminator architecture (e.g., a discriminator architecture from face-vid2vid for training with GAN loss).

[0054] In at least one embodiment, system 400 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 400 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 400 performs... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 400 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0055] Figure 5 The illustration depicts a process 500 using an API to cause a processor to pause instructions, according to at least one embodiment. In at least one embodiment, process 500 includes one or more steps for receiving 502 one or more inputs indicating a context, identifying 504 one or more instructions associated with the context, such that 506 the instruction scheduling is modified to pause execution of one or more identified instructions with a return indication, returning 512 an indication, returning 510 a success indication, and executing... Figure 1-12 The process 500 may be one or more operations, and / or combinations thereof, as described herein. In at least one embodiment, process 500 begins when invoked by one or more processors.

[0056] In at least one embodiment, the processor executes process 500, which includes using an API that receives 502 one or more indications (e.g., identifiers) of one or more contexts as input. For example, the kernel may be suspended, and the context may be indicated to be suspended and / or to be executed in place of or prior to one or more instructions that will be suspended.

[0057] In at least one embodiment, the processor executes process 500, which includes an API for identifying one or more instructions associated with a context, such as by removing these instructions from a queue of instructions to be executed (e.g., a kernel). In at least one embodiment, the processor executes process 500, which includes causing the GPU to suspend execution of indicated instructions corresponding to one or more contexts, and / or the processor executing process 500 to continue causing the instruction schedule to be modified to suspend execution of one or more identified instructions. The system executing process 500 may cause the instruction schedule to be modified to suspend execution of the identified instructions, and / or continue to determine block 508.

[0058] In at least one embodiment, if the processor (e.g., GPU host 202) successfully suspends the indicated instruction and / or executes the second indicated instruction, the decision in decision block 508 is "yes"; otherwise, the decision is "no". In at least one embodiment, if the decision in decision block 508 is "yes", the processor 508 returns one or more success indications, which may include storing a flag in memory. In at least one embodiment, if the decision in decision block 508 is "no", the processor executing process 500 returns one or more indications of failure, error, or variations thereof, for example, by storing a flag in memory.

[0059] In at least one embodiment, the processor executes process 500, which includes using API 1206A (see...). Figure 12 The execution of process 500 is paused to suspend the execution of indicated instructions associated with the context, such as suspending the kernel execution of application 406. In at least one embodiment, the system executing process 500 is described in conjunction with system 400 performing API operations (e.g., API 110 and / or API 1206A). In at least one embodiment, when one or more instructions 510 and / or 512 are returned, the system executing process 500 performs one or more operations described herein, and / or proceeds to completion.

[0060] In at least one embodiment, one or more processors use process 500, such as to execute an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, as an example, a set of instructions is stored on a machine-readable medium (e.g., non-transitory) that, if executed by one or more processors, causes one or more processors to execute process 500, such as to execute an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, process 500 is... Figure 1-12 The process shown is included Figure 1-12 The process shown and / or otherwise includes Figure 1-12 The process shown includes: receiving one or more application programming interface (API) parameters indicating a context for executing GPU instructions on a graphics processing unit (GPU); identifying one or more instructions scheduled for execution on the GPU and associated with the indicated context; and, in response to receiving these parameters, pausing the execution of the one or more instructions and / or otherwise performing the operations described herein. In at least one embodiment, Figure 1-12 The illustrated system execution process 500 includes, for example, executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, Figure 13-31C One or more hardware usage processes 500 shown herein, for example, to execute an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0061] Figure 6This is a block diagram illustrating a system 600 that uses an API to cause a processor to pause an instruction set according to at least one embodiment. In at least one embodiment, system 600 includes one or more APIs 110 (e.g., for pausing 118 one or more instructions associated with an application), one or more processors (e.g., a host GPU 202, see below). Figure 2A-2B ), one or more engines 202 and / or one or more applications 606, one or more instructions (e.g., kernels) executed using one or more contexts, and / or one or more components described herein.

[0062] In at least one embodiment, system 600 executes API 110 (e.g., cuCtxPauseAll()) to pause 118 one or more instructions associated with an application (e.g., application 606). In at least one embodiment, API 110 for pausing 118 one or more instructions associated with the application does not require (but may include) one or more parameters as input 602, such as an indication 604 to the GPU of one or more instructions associated with application 606 that will be paused. In at least one embodiment, when system 600 invokes API 110 (e.g., cuCtxPauseAll()) to pause 118 one or more instructions associated with the application, it pauses all contexts corresponding to all kernels requested by application 606. In at least one embodiment, after API 110 pauses 118 one or more instructions associated with the application, a second API 110 (e.g., for resuming 120 and / or 122 one or more instructions associated with the context and / or application) can be used to resume one or more instructions.

[0063] In at least one embodiment, system 600 executes process 700 to suspend one or more instructions associated with an application using API 110. In at least one embodiment, the output 608 of executing API 110 to suspend one or more instructions associated with the application includes an indication 610 indicating whether the GPU has successfully suspended all instructions in the instruction set. In at least one embodiment, in system 600, the GPU provides output 608 as a response to executing API 110, such as output 608 including one or more success indications 610 (e.g., flags), such as "success," "failure," and / or variations thereof. In at least one embodiment, an example includes suspending contexts 1 to N of a set of contexts for application 606 when API 110 for suspending one or more instructions associated with the application is invoked.

[0064] In at least one embodiment, system 600 includes a collection of one or more hardware and / or software computing resources having instructions that, when executed, perform one or more communication processes, such as those described herein. In at least one embodiment, system 600 is a software program, an application program, and / or variations thereof, executing on computer hardware. In at least one embodiment, one or more processes of system 600 are executed by any suitable processing system or unit as described below (e.g., graphics processing unit (GPU), general-purpose GPU (GPGPU), parallel processing unit (PPU), central processing unit (CPU), data processing unit (DPU)), and in any suitable manner, including sequential execution, parallel execution, and / or variations thereof. In at least one embodiment, system 600 uses a machine learning training framework (e.g., PYTORCH, TENSORFLOW, BOOST, CAFFE, MICROSOFT COGNITIVETOOLKIT / CNTK, MXNET, CHAINER, KERAS, DEEPLEARNING4J, and / or other training frameworks) to implement and perform the operations described herein to select one or more neural networks to perform one or more wireless signal operations. In at least one embodiment, as an example, training the neural network model includes using a server (e.g., an NVIDIA DGX server) that also includes at least a GPU (e.g., AMD MI200, VEGAL10, VEGO20, and ARCTURUS), an optimizer (e.g., ADAM OPTIMIZER), or a discriminator architecture (e.g., a discriminator architecture from face-vid2vid for training using GAN loss).

[0065] In at least one embodiment, system 600 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 600 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 600 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 600 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), may be used to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0066] Figure 7 This is a flowchart illustrating a process 700 of using an API to cause a processor to pause an instruction set according to at least one embodiment. In at least one embodiment, process 700 includes one or more steps for receiving 702 one or more inputs instructing an application, identifying 704 one or more instructions associated with the application, such that 706 instruction scheduling is modified to pause execution of the identified instructions of the application, and executing... Figure 1-12 The process 700 may be one or more operations described herein, and / or combinations thereof. In at least one embodiment, process 700 begins upon being invoked by one or more processors.

[0067] In at least one embodiment, the processor executes process 700, which includes using an API that can receive, 702, one or more instructions 704 of an application whose GPU instructions are to be suspended, one or more instructions to be executed in place of and / or prior to one or more suspended instructions as input, and / or one or more inputs described herein. In one embodiment, the API can also execute without input, simply by invoking it. In this case, 702 may include the API determining the identity of the application invoking the API.

[0068] In at least one embodiment, the processor executes process 700, which includes an API for causing (e.g., pause 118) the GPU to identify 704 one or more instructions associated with application 606, for example, by continuing execution of the next instruction in the instruction queue (e.g., kernel) (e.g., the next corresponding time slice). In at least one embodiment, the processor executes process 700 to identify 704 one or more instructions associated with the application, and / or continues to cause 708 the instruction scheduling to be modified, thereby pausing the execution of one or more instructions of the identified application.

[0069] In at least one embodiment, if the processor (e.g., GPU host 202) successfully suspends the application, the decision in decision box 708 is "yes," otherwise it is "no." In at least one embodiment, if the decision in decision box 708 is "yes," the processor returns one or more success indications in 708, which may include storing a flag in memory. In at least one embodiment, if the decision in decision box 708 is "no," the processor executing process 700 returns one or more indications of failure, error, or variations thereof, for example, by storing a flag in memory.

[0070] In at least one embodiment, the processor executes process 700, which includes using API 1206B (see...). Figure 12 This causes the GPU to pause one or more instructions associated with an application, such as pausing a set of kernels for application 606. In at least one embodiment, the system executing process 700 is described in conjunction with system 600 performing API operations (e.g., API 110 and / or API 1206B). In at least one embodiment, upon return of one or more instructions 712 and / or 710, the system executing process 700 performs one or more operations described herein and / or proceeds to completion.

[0071] In at least one embodiment, one or more processors use process 700 (e.g., a process including one or more operations) to receive an instruction via an application programming interface (API) indicating that the execution of one or more instructions scheduled to be executed on a graphics processing unit (GPU) will be suspended; identify one or more instructions associated with a program that has invoked the API; and, in response to the API invocation, cause the execution of the identified one or more scheduled instructions to be suspended, and / or otherwise perform the operations described herein. In at least one embodiment, as an example, a set of instructions is stored on a machine-readable medium (e.g., non-transitory) that, if executed by one or more processors, causes one or more processors to perform process 700, such as to receive an instruction via an application programming interface (API) indicating that the execution of one or more instructions scheduled to be executed on a graphics processing unit (GPU) will be suspended; identify one or more instructions associated with a program that has invoked the API; and, in response to the API invocation, cause the execution of the identified one or more scheduled instructions to be suspended, and / or otherwise perform the operations described herein. In at least one embodiment, process 700 is Figure 1-12 The process shown is included Figure 1-12 The process shown, and / or otherwise includes Figure 1-12The process described herein involves receiving an instruction via an application programming interface (API) indicating that the execution of one or more instructions scheduled to be executed on a graphics processing unit (GPU) will be suspended; identifying one or more instructions associated with a program that has invoked the API; and, in response to the API invocation, causing the execution of the identified one or more scheduled instructions to be suspended, and / or otherwise performing the operations described herein.

[0072] In at least one embodiment, Figure 1-12 The illustrated system execution process 700 includes, for example, receiving an instruction via an application programming interface (API) indicating that the execution of one or more instructions scheduled to be executed on a graphics processing unit (GPU) will be suspended; identifying one or more instructions associated with a program that has invoked the API; and, in response to the API invocation, causing the execution of the identified one or more scheduled instructions to be suspended, and / or otherwise performing the operations described herein. In at least one embodiment, Figure 13-31C One or more hardware usage processes 700 shown include, for example, receiving an instruction via an application programming interface (API) indicating that the execution of one or more instructions scheduled to be executed on a graphics processing unit (GPU) will be suspended; identifying one or more instructions associated with a program that has invoked the API; and, in response to the API invocation, causing the execution of the identified one or more scheduled instructions to be suspended, and / or otherwise performing the operations described herein.

[0073] Figure 8 This is a block diagram illustrating a system 800 according to at least one embodiment, using an API to cause a processor to resume execution of one or more instructions. In at least one embodiment, system 800 includes one or more APIs 110 (e.g., for resuming one or more instructions associated with a context 120), one or more processors (e.g., host GPU 302, see below) Figures 3A-3B ), one or more engines 302 and / or one or more applications 806, one or more instructions (e.g., kernels) executed using one or more contexts, and / or one or more components described herein.

[0074] In at least one embodiment, system 800 executes API 110 (e.g., cuCtxResumeExecution()) to cause GPU 120 to resume execution of one or more instructions whose execution was previously suspended (e.g., suspended by API 110). In at least one embodiment, the input of system 800 to API 110 for causing GPU 120 to resume execution of one or more instructions (e.g., instructions and / or kernels) is an indication 804 of one or more contexts of one or more applications 806 (e.g., one or more instructions) to be resumed. In at least one embodiment, when system 800 executes API 110 for causing GPU 120 to resume execution of one or more instructions (e.g., kernels), the API causes one or more instructions indicated to the API to be inserted into their previous clock cycle, thereby re-adding them to the run list (e.g., the previous time slice) and / or adding them to the head of the queue of instructions to be executed.

[0075] In at least one embodiment, system 800 executes process 900 (see...) Figure 9 The system 800 uses API 110 to cause GPU 120 to resume execution of instructions (e.g., kernel). In at least one embodiment, the output 808 of executing API 110 to cause GPU 120 to resume execution of instructions (e.g., kernel) includes an indication 810 indicating whether the GPU successfully resumed the instructions. In at least one embodiment, in system 800, the GPU provides output 808 as a response to the execution of API 110, for example, output 808 includes one or more success indications 810 (e.g., flags), such as "success," "failure," and / or variations thereof. In at least one embodiment, an example includes: when API 110 is invoked to resume one or more instructions associated with (e.g., corresponding to) a context, the selected instructions in context 2 of application 806 are resumed.

[0076] In at least one embodiment, system 800 includes a collection of one or more hardware and / or software computing resources having instructions that, when executed, perform one or more communication processes, such as those described herein. In at least one embodiment, system 800 is a software program, an application program, and / or variations thereof, executing on computer hardware. In at least one embodiment, one or more processes of system 800 are executed in any suitable manner by any suitable processing system or unit such as described below (e.g., a graphics processing unit (GPU), a general-purpose GPU (GPGPU), a parallel processing unit (PPU), a central processing unit (CPU), or a data processing unit (DPU)), including sequential execution, parallel execution, and / or variations thereof. In at least one embodiment, system 800 uses a machine learning training framework (e.g., PYTORCH, TENSORFLOW, BOOST, CAFFE, MICROSOFT COGNITIVE TOOLKIT / CNTK, MXNET, CHAINER, KERAS, DEEPLEARNING4J, and / or other training frameworks) to implement and perform the operations described herein to select one or more neural networks to perform one or more wireless signal operations. As an example, in at least one embodiment, training the neural network model includes using a server (e.g., an NVIDIA DGX server) that also includes at least a GPU (e.g., AMD MI200, VEGAL10, VEGO20, and ARCTURUS), an optimizer (e.g., ADAMOPTIMIZER), or a discriminator architecture (e.g., a discriminator architecture from face-vid2vid for training using GAN loss).

[0077] In at least one embodiment, system 800 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more graphics processing unit (GPU) instructions whose execution on the GPU has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 800 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to resume the execution of one or more graphics processing unit (GPU) instructions whose execution on the GPU has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 800 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API) to resume the execution of one or more graphics processing unit (GPU) instructions whose execution on the GPU has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 800 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), to resume the execution of one or more graphics processing unit (GPU) instructions whose execution on the GPU has been suspended, wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0078] Figure 9 This is a flowchart illustrating a process 900 of using an API to cause a processor to resume execution of one or more instructions according to at least one embodiment. In at least one embodiment, process 900 includes one or more steps for receiving one or more inputs indicating a context 902, identifying one or more instructions associated with the context 904, causing the instruction scheduling 906 to be modified to resume execution of one or more identified instructions, returning an indication 912, returning a success indication 910, and executing... Figure 1-12 The process 900 may involve one or more operations described herein, and / or combinations thereof. In at least one embodiment, process 900 begins upon invocation by one or more processors. In at least one embodiment, the processor executes process 900, which includes using an API that receives one or more indications 804 (e.g., identifiers) of context 902 as input.

[0079] In at least one embodiment, the processor executes process 900, which includes API identification 904 (e.g., and causing 120) of one or more instructions associated with a context, such as by returning to an instruction queue (e.g., kernel) to be executed (e.g., executed in the corresponding time slice). The processor may execute process 900, which identifies one or more instructions corresponding to one or more contexts, and / or the processor executing process 900 may continue to cause the instruction scheduling to be modified to resume execution of one or more identified instructions. As an example, one or more schedulers may be used to modify the scheduling. The system may execute process 900 to at least cause the instruction scheduling to be modified to resume execution of one or more identified instructions, and / or continue to decision block 908.

[0080] In at least one embodiment, if the processor (e.g., GPU host 302) successfully recovers the indicated instruction, the decision in decision block 908 is "yes"; otherwise, it is "no". In at least one embodiment, if the decision in decision block 906 is "yes", the processor returns 910 one or more success indications, which may include storing a flag in memory. In at least one embodiment, if the decision in decision block 908 is "no", the processor executing process 900 returns one or more indications of failure, error, or variations thereof, for example, by storing a flag in memory.

[0081] In at least one embodiment, the processor executes process 900, which includes using API 1206C (see...). Figure 12 This is used to resume one or more instructions associated with a context, such as resuming the kernel execution of application 806. In at least one embodiment, the system executing procedure 900 is described in conjunction with executing API operations (e.g., API 110 and / or API 1206C). In at least one embodiment, the system executing procedure 900, after returning one or more instructions 912 and / or 910, performs one or more operations described herein and / or proceeds to completion.

[0082] In at least one embodiment, one or more processors use process 900, for example, a process for: receiving one or more application programming interface (API) parameters indicating the context of graphics processing unit (GPU) instructions on a GPU; identifying one or more instructions suspended on the GPU in relation to the indicated context; and, in response to receiving these parameters, causing the execution of one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein. In at least one embodiment, as an example, a set of instructions is stored on a machine-readable medium (e.g., non-transitory) that, if executed by one or more processors, causes one or more processors to perform process 900, for example, to receive one or more application programming interface (API) parameters indicating the context of graphics processing unit (GPU) instructions on a GPU; identify one or more instructions suspended on the GPU in relation to the indicated context; and, in response to receiving these parameters, causing the execution of one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein. In at least one embodiment, process 900 is... Figure 1-12 The process shown is included Figure 1-12 The process shown and / or otherwise includes Figure 1-12The process shown is used to receive one or more application programming interface (API) parameters indicating the context of graphics processing unit (GPU) instructions on the GPU; identify one or more instructions suspended on the GPU associated with the indicated context; and, in response to receiving these parameters, cause the execution of one or more suspended instructions to be resumed, and / or otherwise perform the operations described herein. In at least one embodiment, Figure 1-12 The illustrated system execution process 900 includes, for example, receiving one or more application programming interface (API) parameters indicating the context of graphics processing unit (GPU) instructions on the GPU; identifying one or more instructions suspended on the GPU associated with the indicated context; and, in response to receiving these parameters, causing the execution of one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein. In at least one embodiment, Figure 13-31C The illustrated hardware usage process 900 includes, for example, receiving one or more application programming interface (API) parameters that indicate the context of graphics processing unit (GPU) instructions on the GPU; identifying one or more instructions that are suspended on the GPU in relation to the indicated context; and, in response to receiving these parameters, causing the execution of one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein.

[0083] Figure 10 This is a block diagram illustrating a system 1000 according to at least one embodiment, using an API to cause a processor to resume execution of one or more instruction sets. In at least one embodiment, system 1000 includes one or more APIs 110 (e.g., for resuming one or more instructions associated with an application 122), one or more processors (e.g., host GPU 302, see...). Figures 3A-3B ), one or more engines 302 and / or one or more applications 1006, one or more instructions (e.g., kernels) executed using one or more contexts, and / or one or more components described herein.

[0084] In at least one embodiment, system 1000 includes API 110 (e.g., cuCtxResumeAll()) for resuming one or more instructions associated with an application, such as all kernels associated with the application. In at least one embodiment, API 110 for resuming one or more instructions associated with the application is executed after API 110 has been used to stop the execution of one or more kernels associated with the application. In at least one embodiment, API 110 does not require (but may include) one or more inputs 1002. As an example, input 1002 may include instruction 1004 from application 1006. In at least one embodiment, when system 1000 executes API 110 to resume one or more instructions associated with the application, the API causes any kernels of the previously suspended application to be added back to the queue of instructions to be executed.

[0085] In at least one embodiment, system 1000 executes process 1100 to recover one or more instructions associated with an application using API 110. In at least one embodiment, the output 1008 of executing API 110 to recover one or more instructions associated with an application includes an indication 1010 indicating whether the GPU has successfully recovered all instructions in the instruction set. In at least one embodiment, in system 1000, the GPU provides output 1008 as a response to executing API 110, for example, output 1008 includes one or more success indications 1010 (e.g., flags), such as "success," "failure," and / or variations thereof. As an example, when API 110 for recovering one or more instructions associated with an application is invoked, the processor can recover any context 1 to N previously suspended for application 1006.

[0086] In at least one embodiment, system 1000 includes a collection of one or more hardware and / or software computing resources having instructions that, when executed, perform one or more communication processes, such as those described herein. In at least one embodiment, system 1000 is a software program executing on computer hardware, an application program running on computer hardware, and / or variations thereof. In at least one embodiment, one or more processes of system 1000 are executed in any suitable manner by any suitable processing system or unit (e.g., graphics processing unit (GPU), general-purpose GPU (GPGPU), parallel processing unit (PPU), central processing unit (CPU), data processing unit (DPU)) as described below, including sequential execution, parallel execution, and / or variations thereof. In at least one embodiment, system 1000 uses a machine learning training framework (e.g., PYTORCH, TENSORFLOW, BOOST, CAFFE, MICROSOFT COGNITIVE TOOLKIT / CNTK, MXNET, CHAINER, KERAS, DEEPLEARNING4J, and / or other training frameworks) to implement and perform the operations described herein to select one or more neural networks to perform one or more wireless signal operations. In at least one embodiment, as an example, training the neural network model includes using a server (e.g., an NVIDIA DGX server) that also includes at least a GPU (e.g., AMD MI200, VEGAL10, VEGO20, and ARCTURUS), an optimizer (e.g., ADAMOPTIMIZER), or a discriminator architecture (e.g., a discriminator architecture from ace-vid2vid for training using GAN loss).

[0087] In at least one embodiment, system 1000 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein the API identifies these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein. In at least one embodiment, system 1000 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein. In at least one embodiment, system 1000 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein. In at least one embodiment, system 1000 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API) to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein.

[0088] Figure 11 This is a flowchart illustrating a process 1100 of using an API to cause a processor to resume execution of one or more instruction sets according to at least one embodiment. In at least one embodiment, process 1100 includes one or more steps for receiving 1102 one or more inputs indicating an application, identifying 1104 one or more instructions associated with the application, causing 1106 instruction scheduling to resume execution of one or more instructions of the identified application, returning 1112 an indication (e.g., error and / or failure), returning 1112 a success indication, and executing... Figure 1-12 The process 1100 may be one or more operations described herein, and / or combinations thereof. In at least one embodiment, process 1100 begins when invoked by one or more processors.

[0089] In at least one embodiment, the processor executes process 1100, which includes using an API that may receive one or more instructions 1004 as input for resuming all instructions in a previously suspended instruction set (e.g., using one or more APIs 110), and / or receive one or more inputs as described herein. In one embodiment, the API may also be executable without input, simply by invocation. In this case, 1102 may include an API for determining the identity of the application invoking the API.

[0090] In at least one embodiment, the processor executes process 1100, which includes API identification 1104 of one or more instructions associated with application 1006 (e.g., by advancing to return said instructions to an instruction queue to be executed (e.g., kernel) (e.g., in a corresponding time slice). In at least one embodiment, the processor executes process 1100 to identify one or more instructions (e.g., context) associated with application 1104, and / or subsequently advances to cause the scheduling of one or more instructions of 1106 to be modified to resume execution of one or more instructions of the identified application. In at least one embodiment, the processor executes process 1100 to cause the scheduling of one or more instructions of 1106 to be modified to resume execution of one or more instructions of the identified application, and / or advances to decision block 1108.

[0091] In at least one embodiment, if the processor (e.g., GPU host 302) successfully recovers all instructions in the instruction set, the decision in decision box 1108 is "yes"; otherwise, the decision is "no". In at least one embodiment, if the decision in decision box 1108 is "yes", the processor returns one or more success indications, which may include storing a flag in memory. In at least one embodiment, if the decision in decision box 1108 is "no", the processor executing process 1100 returns one or more indications of failure, error, or variations thereof, for example, by storing a flag in memory.

[0092] In at least one embodiment, the processor executes process 1100, which includes using API 1206D (see...). Figure 12 This is used to resume the execution of one or more instructions associated with an application, such as resuming the execution of all kernels in a set of kernels for application 1006. In at least one embodiment, the system executing process 1100 is described in conjunction with the execution of API operations 1002 (e.g., API 110 and / or API 1206D). In at least one embodiment, after returning one or more instructions 1110 and / or 1112, the system executing process 1100 performs one or more operations described herein and / or proceeds to completion.

[0093] In at least one embodiment, one or more processors use process 1100, such as a process for: receiving via an application programming interface (API) an indication that execution of one or more suspended instructions of a graphics processing unit (GPU) instruction will be resumed; identifying one or more suspended instructions associated with a program that has invoked the API; and, in response to invoking the API, causing execution of the identified one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein. In at least one embodiment, as an example, a set of instructions is stored on a machine-readable medium (e.g., non-transitory) that, if executed by one or more processors, causes one or more processors to perform process 1100, such as receiving via an application programming interface (API) an indication that execution of one or more suspended instructions of a graphics processing unit (GPU) instruction will be resumed; identifying one or more suspended instructions associated with a program that has invoked the API; and, in response to invoking the API, causing execution of the identified one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein.

[0094] In at least one embodiment, process 1100 is Figure 1-12 The process shown is included Figure 1-12 The process shown and / or otherwise includes Figure 1-12 The process illustrated is used to receive, via an application programming interface (API), an indication that the execution of one or more suspended instructions of a graphics processing unit (GPU) instruction will be resumed; identify one or more suspended instructions associated with a program that has invoked the API; and, in response to invoking the API, cause the execution of the identified one or more suspended instructions to be resumed, and / or otherwise perform the operations described herein. In at least one embodiment, Figure 1-12 The illustrated system execution process 1100 includes, for example, receiving via an application programming interface (API) an indication that the execution of one or more suspended instructions of a graphics processing unit (GPU) instruction will be resumed; identifying one or more suspended instructions associated with a program that has invoked the API; and, in response to invoking the API, causing the execution of the identified one or more suspended instructions to be resumed, and / or otherwise performing the operations described herein. In at least one embodiment, Figure 13-31COne or more hardware usage processes 1100 shown herein include, for example, receiving via an application programming interface (API) an indication that the execution of one or more suspended instructions of a graphics processing unit (GPU) instruction will be resumed; identifying one or more suspended instructions associated with a program that has invoked the API; in response to invoking the API, causing the execution of the identified one or more suspended instructions to be resumed; and / or otherwise performing the operations described herein.

[0095] Figure 12 The illustration depicts a processor 1208 and a storage device 1202 according to at least one embodiment. In at least one embodiment, at least one example of system 1200 includes software and hardware according to at least one embodiment for causing the GPU to pause, cause, and / or resume the execution of one or more instructions (e.g., kernels), or otherwise perform any of the operations described herein. System 1200 may include storage device 1202 and processor 1208. Storage device 1202 may include, for example, memory, cache, or other storage devices further described herein. Storage device 1202 may be separate from processor 1208, or storage device 1202 may be included in processor 1208 (e.g., included in storage device 1212). In at least one embodiment, software program 1204 and / or software library (or instructions) 1206 may be stored in memory, cache, or other storage devices and provided to processor 1208 to cause one or more circuits of processor 1208 to perform the operations described herein. In at least one embodiment, software program 1204 and / or software library (or instructions) 1206 may be integrated into one or more circuits of processor 1208. Software program 1204, which can be used to perform any of the operations described herein, may be stored on storage device 1202.

[0096] In at least one embodiment, the software program 1204 may include one or more software modules. In at least one embodiment, one or more modules include pausing, triggering, and / or resuming the execution of one or more instructions (e.g., kernel and / or instructions).

[0097] In at least one embodiment, as used in any of the embodiments described herein, unless the context explicitly states otherwise, a module refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. In at least one embodiment, software is embodied as a software package, code, and / or instruction set or instructions, and "hardware" as used in any of the embodiments described herein includes (e.g., individually or in any combination) such as hardwired circuitry, programmable circuitry, state machine circuitry, fixed-function circuitry, execution unit circuitry, and / or firmware storing instructions executed by the programmable circuitry. In at least one embodiment, a module may be embodied collectively or individually as a circuitry forming part of a larger system (e.g., an integrated circuit (IC), a system-on-a-chip (SoC), etc.). In at least one embodiment, a module is combined with any suitable processing unit and / or combination of processing units (e.g., one or more CPUs, GPUs, GPGPUs, PPUs, and / or variants thereof, including those further described herein) to perform one or more processes.

[0098] In at least one embodiment, software program 1204 may include a collection of software code, commands, instructions, or other text sequences for instructing a computing device to perform one or more computational operations and / or invoke one or more other instruction sets (e.g., APIs or API functions or Instruction Set Architecture (ISA) level instructions) to be executed or otherwise implemented. Instructions (e.g., hardware instructions) or microcode may include ISA-level instructions, which may include native ISA instructions or non-native ISA instructions. Software program 1204 and / or software library 1206 (e.g., one or more modules) may be distributed across multiple processors that can communicate via buses, networks, writes to shared memory, and / or any suitable communication process (e.g., the communication processes described herein).

[0099] In at least one embodiment, system 1200 may include one or more software libraries 1206, which may provide one or more API and / or ISA instructions. In at least one embodiment, one or more API and / or ISA instructions may be used to cause the GPU to pause, initiate, and / or resume the execution of one or more instructions (e.g., kernels). In at least one embodiment, one or more software libraries 1206 may be included in a driver and / or runtime. In at least one embodiment, software library 1206 (e.g., including one or more API and / or ISA instructions) may include software instruction sets that, if executed or otherwise implemented, cause processor 1208 to perform one or more computational operations, such as any of the operations described herein. In at least one embodiment, one or more API and / or ISA instructions may be distributed or otherwise provided as part of one or more software libraries 1206, runtimes, drivers, and / or any other grouping of software and / or executable code further described herein. In at least one embodiment, one or more API and / or ISA instructions may perform one or more computational operations in response to a call to software program 1204.

[0100] One or more processors 1208 may include any number of processors and any suitable processing units and / or combinations of processing units, such as, but not limited to, a central processing unit (“CPU”), a graphics processing unit (“GPU”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, parallel processors, GPGPUs, DPUs, and / or variations thereof (including those further described herein)). These other processors include any processors described herein, such as, but not limited to, those described herein. Figure 14-26BThe processors in the memory. In at least one embodiment, one or more processors 1208 may retrieve or fetch instructions (e.g., one or more API and / or ISA instructions) from memory 1202 using, for example, instruction fetch 1216 (e.g., for an instruction fetch phase). Instructions may include instructions for causing the GPU to pause, initiate, and / or resume the execution of one or more instructions (e.g., kernels). In at least one embodiment, one or more processors 1208 may include memory 1212 and instruction queue 1210 for storing and queuing instructions fetched from memory 1202. In at least one embodiment, the fetched instructions may be decoded by decoder 1218 to determine what operation one or more processors 1208 should perform (e.g., in an instruction decoding phase). In at least one embodiment, one or more processors 1208 may fetch additional operands (data) that can be used for instructions, and the operands may be stored, for example, in registers or memory 1212. In at least one embodiment, micro-operations 1220 may perform operations on the data stored in one or more registers or memory 1212. For example, each step of an instruction fetched by one or more processors 1208 can be broken down during execution, so that one or more processors 1208 can execute the instruction step by step through a series of micro-operations 1220. In at least one embodiment, the program counter (PC) 1214 can hold the address of the next instruction and can be updated to point to the next instruction to be executed by one or more processors 1208.

[0101] In at least one embodiment, one or more processors 1208 may (e.g., during the execution phase) execute instructions. For example, one or more processors 1208 may execute operations specified by the instructions, such as arithmetic operations, logical operations, or data transfers. In at least one embodiment, one or more computing units 1222 may execute instructions to perform any of the operations described herein. In at least one embodiment, one or more computing units may include one or more ALUs 1224 (Arithmetic Logic Units) that can be used to perform arithmetic and logical operations. In at least one embodiment, one or more computing units may include one or more FPUs (Floating Point Units) 1226 that can be used to perform floating-point calculations. In at least one embodiment, other circuitry 1228 may be used to perform other operations, such as vector operations and / or scalar operations. In at least one embodiment, one or more accelerators 1230 may include one or more matrix multiplication accelerators, one or more parallel processing units (PPUs) (such as GPUs), or any other accelerators or processors further described herein. In at least one embodiment, software program 1204 may utilize one or more API and / or ISA instructions to perform various computational operations, such as matrix multiplication, arithmetic operations, or any other computational operations further described herein, via one or more accelerators 1230. In at least one embodiment, one or more computational operations using one or more accelerators 1230 may include at least one or more sets of computational operations for acceleration by execution at least partially by one or more accelerators 1230, including instructions for causing the GPU to pause, initiate, and / or resume the execution of one or more instructions (e.g., kernels).

[0102] In at least one embodiment, system 1200 can be used to execute one or more instructions, which include functions or operations, such as combinations. Figure 1-12 The functions or operations described herein. In at least one embodiment, system 1200, including one or more processors, causes one or more circuits to cause the GPU to suspend, initiate, and / or resume the execution of one or more instructions (e.g., kernels), and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 is included Figure 1-12 The system shown, and / or otherwise includes Figure 1-12 The system shown is configured to cause one or more circuits to execute APIs, thereby causing the GPU to pause, initiate, and / or resume the execution of one or more instructions (e.g., kernels), and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 includes... Figure 1-12The hardware shown includes one or more hardware components, such as those used to execute APIs to cause the GPU to pause, initiate, and / or resume the execution of one or more instructions (e.g., kernels), and / or otherwise perform the operations described herein.

[0103] In at least one embodiment, system 1200 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 performs... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), may be used to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0104] In at least one embodiment, system 1200 includes one or more processors configured to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 is Figure 1-12 The system shown is included Figure 1-12 The system shown or otherwise includes Figure 1-12The system shown is used to execute an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as associated with and / or otherwise perform the operations described herein by a program that has invoked the API. In at least one embodiment, system 1200 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), to suspend the execution of one or more GPU instructions scheduled to execute on a graphics processing unit (GPU), wherein these GPU instructions are recognized by the API as associated with a program that has invoked the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API), may be used to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein these GPU instructions are identified by the API as being associated with a program that has invoked the API, and / or otherwise perform the operations described herein.

[0105] In at least one embodiment, system 1200 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein these GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is configured to execute an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein these GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API), to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein these GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein. In at least one embodiment, system 1200 includes Figure 13-31CThe hardware shown may be used, for example, to execute an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein these GPU instructions are associated with one or more identifiers indicated to the API, and / or otherwise perform the operations described herein.

[0106] In at least one embodiment, system 1200 includes one or more processors configured to execute an application programming interface (API) to resume execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein. In at least one embodiment, system 1200 is Figure 1-12 The system shown is included Figure 1-12 The system shown and / or otherwise includes Figure 1-12 The system shown is used to execute an application programming interface (API) to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein. In at least one embodiment, system 1200 executes... Figure 1-12 The illustrated one or more processes, such as executing an application programming interface (API) to resume the execution of one or more GPU instructions whose execution on a graphics processing unit (GPU) has been suspended, wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein. In at least one embodiment, system 1200 includes Figure 13-31C One or more pieces of hardware, such as those used to execute an application programming interface (API) to resume the execution of one or more GPU instructions that have been suspended on a graphics processing unit (GPU), wherein the API recognizes these GPU instructions as associated with a program that has invoked the API, and / or otherwise performs the operations described herein.

[0107] Various techniques have been described in the preceding and following sections. For ease of explanation, specific configurations and details have been elaborated to provide a comprehensive understanding of the possible ways to implement these techniques. However, it is clear that the techniques described below can also be implemented in different configurations without requiring specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the techniques being described.

[0108] Data Center Figure 13An example data center 1300 according to at least one embodiment is illustrated. The data center 1300 may include one or more rooms having racks 1302 and auxiliary equipment for housing one or more racks 1302 and one or more substrates 1304. Racks 1302 may include one or more substrates 1304. Racks 1302 may include enclosures for housing and supporting individual substrates 1304. Operational aspects of racks 1302 may be adjustable at the rack level (corresponding to a group of substrates 1304) or at the substrate level (corresponding to an individual substrate 1304), among other options. Racks 1302 or substrates 1304 may have specific selected maximum operating parameters, such as, but not limited to, power consumption, operating frequency, etc. The data center 1300 may be supported by various cooling systems, such as, but not limited to, cooling towers, cooling loops, pumps, and other support systems. The cooling system may include sensors and controllers for monitoring and managing the cooling characteristics of racks 1302. The substrates 1304 within racks 1302 may draw operating power from one or more power distribution units (PDUs; not shown). PDUs can be arranged within racks 1302, for example, between racks 1302 that include substrates 1304, or within racks 1302 that also house substrates 1304.

[0109] Rack 1302 and substrate 1304 may include subsystems, modules, add-in cards, and other semiconductor components. Substrate 1304 may include one or more computing units 1306, each computing unit 1306 may include one or more processors 1308, one or more memories 1310, and an interface controller 1312. Computing unit 1306 may include any number of processors, such as, but not limited to, a central processing unit (“CPU”), a graphics processing unit (“GPU”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), including any processors described herein, such as, but not limited to, those described herein. Figure 11-23 The processor in B. The computing unit 1306 may include one or more memory storage devices 1310 (e.g., dynamic read-only memory, solid-state storage devices, or disk drives), as well as network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power supply modules, and cooling modules, etc. One or more computing units 1306 may be a server having one or more of the aforementioned computing resources.

[0110] Computing unit 1306 may include individual computing unit groups housed in one or more racks (not shown), or in numerous racks located in data centers (also not shown) at different geographical locations. Individual computing unit groups may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. Several computing units (e.g., including CPUs and / or other processors) may be grouped within one or more racks to provide computing resources to support one or more workloads. Resource coordinator 1314 may configure or otherwise control one or more computing units 1306 or computing unit groups. Resource coordinator 1314 may include a Software Design Infrastructure (“SDI”) management entity for data center 1300. Resource coordinator 1314 may include hardware, software, or some combination thereof.

[0111] Data center 1300 may include any one or any combination of the framework layer 1320, software layer 1330, and application layer 1340. For example... Figure 13 As shown, framework layer 1320 includes job scheduler 1322, configuration manager 1324, resource manager 1326, and distributed file system 1328. Framework layer 1320 may include a framework for supporting software 1332 of software layer 1330 and / or one or more applications 1342 of application layer 1340. Software 1332 or application 1342 may respectively include web-based service software or applications, such as, but not limited to, software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1320 may be a type of free and open-source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), which can leverage distributed file system 1328 for large-scale data processing (e.g., “big data”). Job scheduler 1322 may include Spark drivers, which facilitate the scheduling of workloads supported by the various layers of data center 1300. Configuration manager 1324 can configure different layers, such as, but not limited to, software layer 1330 and framework layer 1320 (which includes Spark and distributed file system 1328 for supporting large-scale data processing). Resource manager 1326 can manage clustered or grouped compute units 1306 mapped to or allocated to support distributed file system 1328 and job scheduler 1322. Resource manager 1326 can coordinate with resource coordinator 1314 to manage these mapped or allocated compute resources.

[0112] Software 1332 may be included in software layer 1330, and may include software used by at least a portion of computing units 1306, one or more computing units 1306, groups of computing units 1306, and / or the distributed file system 1328 of framework layer 1320. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0113] Application 1342 may be included in application layer 1340 and may include one or more types of applications used by at least portions of computing unit 1306, one or more computing units 1306, groups of computing units 1306, and / or the distributed file system 1328 of framework layer 1320. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing applications, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0114] Any of the Configuration Manager 1324, Resource Manager 1326, and Resource Coordinator 1314 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can alleviate the burden on data center operators of Data Center 1300 to make potentially erroneous configuration decisions and may prevent underutilized and / or poorly performing portions of the data center.

[0115] Data center 1300 may include tools, services, software, or other resources for training one or more machine learning models according to one or more embodiments described herein, or for using one or more machine learning models to predict or infer information. For example, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above regarding data center 1300. The trained machine learning model corresponding to one or more neural networks can be used with the resources described above regarding data center 1300 to infer or predict information using weight parameters calculated through one or more training techniques described herein.

[0116] Data Center 1300 can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware (e.g., Figure 14-26BThe embodiments described herein can be used to perform some or all of the processes and techniques described elsewhere, such as, but not limited to, training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow a user to train or perform information inference, such as, but not limited to, image recognition, speech recognition, or other artificial intelligence services.

[0117] In at least one embodiment, processor 1308 may include one of the processors described below and / or include one or more circuitry for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. In at least one embodiment, processor 1308 is configured by software 1332 to execute an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. Data center 1300 may use logic, CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware (e.g., Figure 14-26B (In the embodiments) perform any of the operations described above or elsewhere in this document.

[0118] processor The following figures illustrate, but are not limited to, example processors and processing systems that can be used to execute application programming interfaces (APIs) to pause and / or cause one or more indicated instructions or otherwise perform some or all of the processes, operations, and / or techniques described elsewhere herein. The example processors and processing systems can be software-configured to execute application programming interfaces (APIs) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. Processors and processing systems may include logic, central processing units (CPUs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), XPUs (i.e., any computing architecture best suited to the needs of the application), or other hardware (e.g., Figure 14-26BThe embodiments described herein are used to perform any of the operations described above, below, or elsewhere herein. The processor and / or processing system described herein may include one or more circuits that can be used to execute an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. As used herein, one or more circuits may be software-configured to execute an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. Figure 31A and Figure 31B The illustration depicts logic 2815 according to at least one embodiment, which, as described elsewhere herein, can be used in one or more devices to perform operations such as, but not limited to, those discussed herein. For example, logic can refer to any combination of software logic, hardware logic, and / or firmware logic that provides the functionality and / or operations described herein, wherein the logic can be collectively or individually embodied as part of a circuit system forming a larger system, such as an integrated circuit (IC), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), or one or more processors (e.g., CPU, GPU).

[0119] Figure 14 A processor according to at least one embodiment is illustrated, which is a system-on-a-chip (SOC) 1400 (which may be referred to as a system-on-a-chip, superchip, or other names). SOC 1400 may include processor complex 1410 and processor complex 1440. SOC 1400 may include any number of processor complexes 1410 and / or processor complexes 1440, which may include any number of processors described herein in any combination, such as, but not limited to, processors described herein in any combination. Figure 14-26B The processor in the system. For example, processor 1410 may include a central processing unit (CPU), and processor 1440 may include a graphics processor. Alternatively, processor 1410 may include a graphics processor, and processor 1440 may include a graphics processor. The SOC 1400 may include any number of display controllers 1492, any number of multimedia engines 1494, any number of I / O interfaces 1470, any number of memory controllers 1480, and any number of fabrics 1460 in any combination. For ease of explanation, this document uses reference numbers identifying objects and bracket numbers identifying instances (if needed) to denote multiple instances of similar objects. The SOC 1400 may include a processor from Broadcom Corporation, Palo Alto, California.

[0120] Processor complex 1410 may include a CPU, processor complex 1440 may include a GPU, and SOC 1400 may include a processing unit integrating processor complex 1410 and processor complex 1440 onto a single chip. Certain instructions may be assigned to processor complex 1410, while other instructions may be assigned to processor complex 1440. Processor complex 1410 may be configured to execute main control software associated with SOC 1400, such as, but not limited to, an operating system. Processor complex 1410 may be the main processor of SOC 1400, controlling and coordinating the operation of other processors. Processor complex 1410 may issue commands that control the operation of processor complex 1440 to perform some or all of the operations described herein. Processor complex 1410 may be configured to execute host-executable code derived from CUDA or other source code (e.g., HIP source code), while processor complex 1440 may be configured to execute device-executable code derived from CUDA or other source code to perform any of the operations described herein.

[0121] Processor complex 1410 may include cores 1420(1)-1420(4) and cache (e.g., L3 cache) 1430 for storing information for performing the operations described herein. Processor complex 1410 may include any number of cores 1420 in any combination and any number and type of cache. Cores 1420 may be configured to execute instructions of a particular instruction set architecture (“ISA”) to perform some or all of the operations described herein. Each core 1420 may include a CPU core. Cores 1420(1)-1420(4) may be referred to as compute units or arithmetic units. SOC 1400 may include any number of processor complexes 1410, architecture 1460, I / O interface 1470, and memory controller 1480.

[0122] Each core 1420 may include a fetch / decode unit 1422, an integer execution engine 1424, a floating-point execution engine 1426, and an L2 cache 1428. The fetch / decode unit 1422 may fetch instructions to perform some or all of the operations described herein (e.g., but not limited to APIs compiled into instructions) and decode those instructions, generate micro-operations, and dispatch individual micro-instructions to the integer execution engine 1424 and / or the floating-point execution engine 1426. The fetch / decode unit 1422 may concurrently dispatch one micro-instruction to the integer execution engine 1424 and another micro-instruction to the floating-point execution engine 1426. The integer execution engine 1424 may perform integer and memory operations. The floating-point engine 1426 may perform floating-point and vector operations. The fetch / decode unit 1422 may dispatch micro-instructions to one or more execution engines, which may replace both the integer execution engine 1424 and the floating-point execution engine 1426.

[0123] Each core 1420(i) (where i is an integer representing a specific instance of core 1420) can access the L2 cache 1428(i) included in core 1420(i). Each core 1420 included in core complex 1410(j) (where j is an integer representing a specific instance of core complex 1410) can be connected to other cores 1420 included in core complex 1410(j) via the L3 cache 1430(j) included in core complex 1410(j). The cores 1420 included in core complex 1410(j) (where j is an integer representing a specific instance of core complex 1410) can access all L3 caches 1430(j) included in core complex 1410(j). The L3 cache 1430 can include any number of slices.

[0124] Processor complex 1440 may be a graphics complex that can be configured to perform computational operations (e.g., the computational operations described herein) in a highly parallel manner. Processor complex 1440 may be configured to perform graphics pipeline operations, such as, but not limited to, drawing commands, pixel operations, geometric calculations, and other operations associated with rendering an image to a display. Processor complex 1440 may be configured to perform graphics-independent operations, such as, but not limited to, neural network training and / or simulation. Processor complex 1440 may be configured to perform both graphics-related and graphics-independent operations.

[0125] Processor complex 1440 may include any number of compute units 1450(1)-1450(N) (where N is any integer greater than 1) and L2 cache 1442. The compute units 1450 may share L2 cache 1442, which may store information that will be used to perform some or all of the operations described herein. L2 cache 1442 may be partitioned. Processor complex 1440 may include any number of compute units 1450 and any number (including zero) and type of cache. Processor complex 1440 may include any number of dedicated graphics hardware.

[0126] Each compute unit 1450 may include any number of SIMD units 1452(1)-1452(N) (where N is any integer greater than 1) and shared memory 1454. Each SIMD unit 1452 may implement a SIMD architecture and may be configured to perform some or all of the operations described herein in parallel. Each compute unit 1450 may execute any number of thread blocks, but each thread block may execute on a single compute unit 1450, although in some embodiments, the thread block may execute on multiple compute units. A thread block may include any number of execution threads. A workgroup may be a thread block. Each SIMD unit 1452 may execute a set of threads. A set of threads (e.g., 16 threads), also referred to as a warp, subgroup, or wavefront (e.g., used by AMD and Intel), may belong to a single thread block and be configured to process different datasets based on a single instruction set. Prediction may be used to disable one or more threads in a warp, subgroup, or wavefront. A lane may be a thread. A work item can be a thread, such as (but not limited to) an OpenCL thread. Different thread bundles, subgroups, or wavefronts within a thread block can be synchronized together and communicate via shared memory 1454. Each compute unit 1450 can include one or more thread block clusters, where thread block clusters can implement programmable control over locality at a larger granularity than a single thread block in a single streaming multiprocessor (SM). Thread block clusters (also referred to as “clusters”) can support multiple thread blocks running concurrently across streaming multiprocessors, thereby synchronously and cooperatively acquiring, exchanging, or otherwise using data. In at least one embodiment, a streaming multiprocessor (“SM”) can refer to a streaming microprocessor, a streaming processor (“SP”), a streaming processing unit (“SPU”), a compute unit (“CU”), an execution unit (“EU”), and / or a slice, where a slice in this context can refer to a portion of the processing resources within a processing unit (e.g., 16 cores, a ray tracing unit, a thread bootstrap, or a scheduler).

[0127] Structure 1460 may be a system interconnect that facilitates data and control transfers across processor complex 1410, processor complex 1440, I / O interface 1470, memory controller 1480, display controller 1492, and multimedia engine 1494, for example, to perform some or all of the operations described herein. SOC 1400 may include any number and type of system interconnects other than or replacing structure 1460, facilitating data and control transfers across any number and type of directly or indirectly linked components within or outside SOC 1400. I / O interface 1470 may represent any number and type of I / O interfaces (e.g., PCI, PCI extensions (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices may be coupled to I / O interface 1470. Peripheral devices that may be coupled to I / O interface 1470 may include keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, etc.

[0128] Display controller 1492 can display images on one or more display devices, such as, but not limited to, liquid crystal displays (“LCD” devices. Multimedia engine 1494 can include any number and type of circuitry related to multimedia, such as, but not limited to, video decoders, video encoders, image signal processors, etc. Memory controller 1480 can facilitate data transfer between SOC 1400 and unified system memory 1490. Processor complex 1410 and processor complex 1440 can share unified system memory 1490. Unified system memory 1490 can include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as, but not limited to, synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Unified system memory 1490 can include 3D stacked memory, including but not limited to high bandwidth memory (HBM), HBM2e, or HDM3.

[0129] The SOC 1400 can implement a memory subsystem including any number and type of memory controllers 1480 and memory devices (e.g., shared memory 1454), which may be dedicated to a single component or shared among multiple components to perform any of the operations described herein. The SOC 1400 can implement a cache subsystem including one or more cache memories (e.g., L2 cache 1428, L3 cache 1430, and L2 cache 1442), each cache memory may be dedicated to any number of components (e.g., core 1420, core complex 1410, SIMD unit 1452, compute unit 1450, and processor complex 1440), or may be shared among any number of components (e.g., core 1420, core complex 1410, SIMD unit 1452, compute unit 1450, and processor complex 1440).

[0130] In at least one embodiment, the SOC 1400 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0131] Figure 15A A parallel processor 1500 according to at least one embodiment is illustrated. The parallel processor 1500 may be implemented using one or more circuits and may be referred to as a programmable processor (e.g., CPU and / or GPU), logic, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other hardware (e.g., Figure 14-26B (The embodiments in the document) are used to perform any of the operations described above or elsewhere herein.

[0132] Parallel processor 1500 may include parallel processing unit 1502 for performing any of the operations described above or elsewhere herein. Parallel processing unit 1502 may include I / O unit 1504 that enables communication with other devices, including other instances of parallel processing unit 1502. I / O unit 1504 may be directly connected to other devices. I / O unit 1504 may be connected to other devices via the use of a hub or switch interface, such as, but not limited to, memory hub 1505. The connection between memory hub 1505 and I / O unit 1504 may form a communication link 1513. I / O unit 1504 may be connected to host interface 1506 and memory crossbar switch 1516, wherein host interface 1506 receives commands directed to perform processing operations, and memory crossbar switch 1516 receives commands directed to perform memory operations.

[0133] When host interface 1506 receives a command buffer via I / O unit 1504, host interface 1506 can route the work operations that execute these commands to front-end 1508. Front-end 1508 can be coupled to scheduler 1510 (which may be referred to as sequencer), which is configured to distribute commands or other work items to processing cluster array 1512. Scheduler 1510 can ensure that processing cluster array 1512 is correctly configured and in an active state before distributing instructions to the cluster of processing cluster array 1512. Scheduler 1510 can be implemented via firmware logic executed on a microcontroller. The microcontroller-implemented scheduler 1510 can be configured to perform complex scheduling and work distribution operations at both coarse and fine granular levels, thereby enabling fast preemption and context switching of threads executing on processing array 1512. Host software can validate workloads scheduled on processing cluster array 1512 via one of multiple graphics processing paths. The workload can then be automatically distributed to the processing array cluster 1512 by the scheduler 1510 logic within the microcontroller, which includes the scheduler 1510.

[0134] Processing cluster array 1512 can perform any of the operations described above or elsewhere herein, and may include up to “N” processing clusters (e.g., clusters 1514A, 1514B through 1514N), where “N” represents a positive integer (which may be a different integer “N” than used in other diagrams). Each cluster 1514A-1514N in processing cluster array 1512 can execute a large number of concurrent threads. Scheduler 1510 may use various scheduling and / or work distribution algorithms to distribute work to clusters 1514A-1514N in processing cluster array 1512, which may vary depending on the workload generated by each type of program or computation. Scheduling may be dynamically handled by scheduler 1510 or may be assisted by compiler logic during the compilation of program logic configured to be executed by processing cluster array 1512. Different clusters 1514A-1514N of processing cluster array 1512 may be assigned to process different types of programs or perform different types of computations.

[0135] The processing cluster array 1512 can be configured to perform various types of parallel processing operations, such as, but not limited to, any of the operations described above or elsewhere herein. The processing cluster array 1512 can be configured to perform general-purpose parallel computing operations. For example, the processing cluster array 1512 may include logic for executing processing instructions, including filtering video and / or audio data, performing modeling operations (including physical operations), and performing data transformations.

[0136] Processing cluster array 1512 can be configured to perform parallel graphics processing operations. Processing cluster array 1512 may include additional logic for supporting the execution of such graphics processing operations, including but not limited to texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. Processing cluster array 1512 can be configured to execute shader programs related to graphics processing, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. Parallel processing unit 1502 can transfer data from system memory via I / O unit 1504 for processing. During processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 1522) during processing and then written back to system memory.

[0137] When the parallel processing unit 1502 is used to perform graphics processing, the scheduler 1510 can be configured to divide the processing workload into instructions of approximately equal size to better distribute graphics processing operations to multiple clusters 1514A-1514N of the processing cluster array 1512. Each part of the processing cluster array 1512 can be configured to perform different types of processing. For example, a first part can be configured to perform vertex shading and topology generation, a second part can be configured to perform tessellation and geometry shading, and a third part can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. Intermediate data generated by one or more clusters 1514A-1514N can be stored in a buffer to allow intermediate data to be transferred between clusters 1514A-1514N for further processing.

[0138] Processing cluster array 1512 can receive processing instructions to be executed via scheduler 1510, which receives commands defining the processing instructions from front end 1508. Processing instructions may include indexes of data to be processed, such as surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands defining how to process the data (e.g., which program to execute). Scheduler 1510 can be configured to retrieve the index corresponding to the instruction, or can receive the index from front end 1508. Front end 1508 can be configured to ensure that processing cluster array 1512 is configured to be active before the workload specified by the incoming command buffer (e.g., batch buffer, push buffer, etc.) is initiated.

[0139] Each instance of one or more instances of parallel processing unit 1502 may be coupled to parallel processor memory 1522 to perform any of the operations described above or elsewhere herein. Parallel processor memory 1522 may be accessed via memory crossbar switch 1516, which may receive memory requests from processing cluster array 1512 and I / O unit 1504. Memory crossbar switch 1516 may access parallel processor memory 1522 via memory interface 1518. Memory interface 1518 may include multiple partition units (e.g., partition units 1520A, 1520B through 1520N), each partition unit may be coupled to a portion (e.g., a memory cell) of parallel processor memory 1522. The number of partition units 1520A-1520N can be configured to be equal to the number of memory units, such that the first partition unit 1520A has a corresponding first memory unit 1524A, the second partition unit 1520B has a corresponding memory unit 1524B, and the Nth partition unit 1520N has a corresponding Nth memory unit 1524N. The number of partition units 1520A-1520N may not be equal to the number of memory units.

[0140] Memory cells 1524A-1524N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as, but not limited to, synchronous graphics random access memory (SGRAM), which includes graphics double data rate (GDDR) memory. Memory cells 1524A-1524N may also include 3D stacked memory, including but not limited to high-bandwidth memory (HBM), HBM2e, or HDM3. Render targets (e.g., but not limited to framebuffers or texture maps) may be stored in memory cells 1524A-1524N, allowing partitioning cells 1520A-1520N to write portions of each render target in parallel to efficiently utilize the available bandwidth of the parallel processor memory 1522. A local instance of the parallel processor memory 1522 may not be included to support a unified memory design that combines system memory with local cache memory.

[0141] Any cluster 1514A-1514N in the processing cluster array 1512 can process data to be written to any memory cell 1524A-1524N within the parallel processor memory 1522. The memory crossbar switch 1516 can be configured to transfer the output of each cluster 1514A-1514N to any partition cell 1520A-1520N, or to another cluster 1514A-1514N on which additional processing operations can be performed. Each cluster 1514A-1514N can communicate with the memory interface 1518 via the memory crossbar switch 1516 to read from or write to various external memory devices. The memory crossbar switch 1516 can be connected to the memory interface 1518 to communicate with the I / O unit 1504, or to a local instance of the parallel processor memory 1522, enabling processing units within different processing clusters 1514A-1514N to communicate with system memory or other memory local to the non-parallel processing unit 1502. The memory crossbar switch 1516 can use virtual channels to separate traffic flows between clusters 1514A-1514N and partition units 1520A-1520N.

[0142] Multiple instances of the parallel processing unit 1502 can be mounted on a single add-in card, or multiple add-in cards can be interconnected. Even if different instances of the parallel processing unit 1502 have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences, these different instances can be configured to interoperate. For example, some instances of the parallel processing unit 1502 may include higher-precision floating-point units relative to other instances. Systems including one or more instances of the parallel processing unit 1502 or the parallel processor 1500 can be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0143] Figure 15A It also includes a block diagram of a partitioning unit 1520 according to at least one embodiment. The partitioning unit 1520 is... Figure 15AAn example of one of the partition units 1520A-1520N in the parallel processor memory. Partition unit 1520 may include an L2 cache 1521, a frame buffer interface 1525, and a ROP 1526 (raster operation unit). The L2 cache 1521 may be a read / write cache configured to perform load and store operations received from the memory crossbar switch 1516 and the ROP 1526. Read misses and urgent write-back requests may be output from the L2 cache 1521 to the frame buffer interface 1525 for processing. Updates may also be sent to the frame buffer via the frame buffer interface 1525 for processing. The frame buffer interface 1525 may interface with one of the memory cells in the parallel processor memory, such as, but not limited to, Figure 15A The memory cells 1524A-1524N (shown as 1524) are located in the parallel processor memory 1522.

[0144] ROP 1526 can be a processing unit that performs raster operations, such as, but not limited to, stenciling, z-testing, blending, etc. ROP 1526 can then output processed graphics data stored in graphics memory. ROP 1526 may include compression logic for compressing depth or color data written to memory and decompressing depth or color data read from memory. The compression logic can be lossless compression logic that utilizes one or more compression algorithms. The type of compression performed by ROP 1526 can vary based on the statistical characteristics of the data to be compressed. For example, incremental color compression is performed on depth and color data on a per-tile basis.

[0145] ROP 1526 can be included in each processing cluster (e.g., Figure 15A The data is stored within clusters 1514A-1514N, not within partition unit 1520. Read and write requests for pixel data (not pixel fragment data) can be transferred via memory crossbar switch 1516. Processed graphics data can be displayed on the monitor and routed for further processing by the processor, or routed to... Figure 15A One of the 1500 parallel processing entities in the process is further processed.

[0146] In at least one embodiment, the parallel processor 1500 may include one or more circuits configured to execute an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. The one or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0147] Figure 15B A block diagram including a processing cluster 1514 within a parallel processing unit according to at least one embodiment. The processing cluster may be... Figure 15A An instance of one of the processing clusters 1514A-1514N is provided, which can be used to perform any of the operations described above or elsewhere herein. Processing cluster 1514 can be configured to execute many threads in parallel, where a “thread” refers to an instance of a specific program executed on a specific input dataset. Single Instruction Multiple Data (SIMD) instruction issuing techniques can be used to support the parallel execution of a large number of threads without providing multiple independent instruction units. Single Instruction Multiple Thread (SIMT) techniques can be used to support the parallel execution of a large number of typically synchronous threads using a common instruction unit configured to issue instructions to a set of processing engines within each processing cluster.

[0148] The operation of cluster 1514 can be controlled via pipeline manager 1532, which distributes processing instructions to the SIMT parallel processors. Pipeline manager 1532 can... Figure 15A The scheduler 1510 receives instructions and manages the execution of these instructions via the graphics multiprocessor 1534 and / or texture unit 1536. The graphics multiprocessor 1534 may be an example instance of a SIMT parallel processor. However, the processing cluster 1514 may include various types of SIMT parallel processors with different architectures. The processing cluster 1514 may include one or more instances of the graphics multiprocessor 1534. The graphics multiprocessor 1534 can process data and can use the data cross switch 1540 to distribute the processed data to one of several possible destinations, including other shader units. The pipeline manager 1532 can facilitate the distribution of processed data by specifying the destination of the processed data to be distributed via the data cross switch 1540.

[0149] Each graphics multiprocessor 1534 within the processing cluster 1514 may include a set of identical functional execution logic (e.g., arithmetic logic units, load-memory units, etc.) for performing computations for any of the operations described above or elsewhere herein. The functional execution logic can be configured in a pipelined manner, where new instructions can be issued before previous instructions complete. The functional execution logic can support a wide range of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and computation of various algebraic functions. Different operations can be performed using the same functional unit hardware, and arbitrary combinations of functional units are possible.

[0150] Instructions transmitted to the processing cluster 1514 can form threads, which may also be called thread bundles, subgroups, waves, or wavefronts. A group of threads executing across a set of parallel processing engines can be called a thread group. Thread groups can execute a common program on different input data. Each thread within a thread group can be assigned to a different processing engine within the graphics multiprocessor 1534. The number of threads in a thread group can be less than the number of processing engines within the graphics multiprocessor 1534. When the number of threads in a thread group is less than the number of processing engines, one or more processing engines may be idle during the processing cycle of that thread group. The number of threads in a thread group can also be more than the number of processing engines within the graphics multiprocessor 1534. When the number of threads in a thread group is more than the number of processing engines within the graphics multiprocessor 1534, processing can be performed in consecutive clock cycles. Multiple thread groups can execute concurrently on the graphics multiprocessor 1534.

[0151] The graphics multiprocessor 1534 includes an internal cache memory for performing load and store operations, such as, but not limited to, any of the operations described above or elsewhere herein. The graphics multiprocessor 1534 may forgo the internal cache and instead use a cache memory within the processing cluster 1514 (e.g., L1 cache 1548). Each graphics multiprocessor 1534 may also access partition units that can be shared across all processing clusters 1514 (e.g., ...). Figure 15A The graphics multiprocessor 1534 has an L2 cache within partition units 1520A-1520N and can be used to transfer data between threads. The graphics multiprocessor 1534 can also access off-chip global memory, which may include one or more of the local parallel processor memory and / or system memory. Any memory outside of the parallel processing unit 1502 can be used as global memory. The processing cluster 1514 may include multiple instances of the graphics multiprocessor 1534 and can share common instructions and data, which can be stored in the L1 cache 1548.

[0152] Each processing cluster 1514 may include an MMU 1545 (Memory Management Unit), which can be configured to map virtual addresses to physical addresses. One or more instances of the MMU 1545 may reside in... Figure 15AThe MMU 1545 is located within the memory interface 1518. It may include: a set of page table entries (PTEs) for mapping virtual addresses to physical addresses of tiles; and optional cache line indexes. The MMU 1545 may include address translation backstop (TLB) buffers or caches that may reside within the graphics multiprocessor 1534 or L1 1548 cache or processing cluster 1514. Physical addresses can be processed to distribute surface data access locally, allowing for efficient request interleaving between partition units. The cache line indexes can be used to determine whether a request for a cache line is a hit or a miss.

[0153] Processing cluster 1514 can be configured such that each graphics multiprocessor 1534 is coupled to a texture unit 1536 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. Texture data can be read from an internal texture L1 cache (not shown) or an L1 cache within the graphics multiprocessor 1534, and can be retrieved as needed from an L2 cache, local parallel processor memory, or system memory. Each graphics multiprocessor 1534 can output processed instructions to a data crossbar switch 1540 to provide the processed instructions to another processing cluster 1514 for further processing, or store the processed instructions in an L2 cache, local parallel processor memory, or system memory via a memory crossbar switch 1516. Pre-ROP 1542 (pre-raster operation unit) can be configured to receive data from the graphics multiprocessor 1534 and direct the data to ROP units, which can be associated with partitioning units described herein (e.g., Figure 15A The PreROP 1542 unit is located together with partition units 1520A-1520N. The PreROP 1542 unit can perform color blending optimization, organize pixel color data, and perform address translation.

[0154] In at least one embodiment, the processing cluster 1514 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0155] Figure 15CA graphics multiprocessor 1534 according to at least one embodiment is illustrated, for example, to perform any of the operations described above or elsewhere herein. The graphics multiprocessor 1534 may be coupled to a pipeline manager 1532 of a processing cluster 1514. The graphics multiprocessor 1534 may include an execution pipeline including, but not limited to, an instruction cache 1552 (e.g., which may store instructions, such as, but not limited to, compiled API instructions), an instruction unit 1554, an address mapping unit 1556, a register file 1558, one or more general-purpose graphics processing unit (GPGPU) cores 1562, and one or more load / store units 1566, one or more of which may perform load / store operations to load / store instructions corresponding to the execution operations. The GPGPU cores 1562 and load / store units 1566 may be coupled to a cache memory 1572 and a shared memory 1570 via a memory and cache interconnect 1568. The GPGPU cores 1562 may be part of a SoC, such as, but not limited to, [other components]. Figure 14 It is part of the integrated circuit 1400.

[0156] Instruction cache 1552 can receive a stream of instructions to be executed (e.g., perform any operation described above or elsewhere herein) from pipeline manager 1532. Instructions can be cached in instruction cache 1552 and dispatched for execution by instruction unit 1554. Instruction unit 1554 can dispatch instructions into thread groups (e.g., thread bundles, subgroups, wavefronts, or waves), with each thread in the thread group assigned to a different execution unit within GPGPU core 1562. Instructions can access any of the local, shared, or global address spaces by specifying an address within a unified address space. Address mapping unit 1556 can be used to translate addresses in the unified address space into different memory addresses accessible by load / store unit 1566.

[0157] Register file 1558 provides a set of registers for the functional units of graphics multiprocessor 1534. Register file 1558 provides temporary storage for operands on data paths connected to functional units of graphics multiprocessor 1534 (e.g., GPGPU core 1562, load / store unit 1566). Register file 1558 can be partitioned among functional units such that each functional unit is allocated a dedicated portion of register file 1558. Register file 1558 can be partitioned among different thread bundles (which may be referred to as wavefronts, subgroups, and / or waves or threads) executed by graphics multiprocessor 1534.

[0158] Each GPGPU core 1562 may include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) for executing instructions of the graphics multiprocessor 1534. The architectures of the GPGPU cores 1562 may be similar or different. A first part of the GPGPU core 1562 may include a single-precision FPU and an integer ALU, while a second part of the GPGPU core may include a double-precision FPU. The FPU may implement IEEE 754-2008 standard floating-point arithmetic or enable variable-precision floating-point arithmetic. The graphics multiprocessor 1534 may also include one or more fixed-function or special-function units for performing specific functions, such as, but not limited to, copying rectangles or pixel blending operations. One or more of the GPGPU cores 1562 may also include fixed-function or special-function logic.

[0159] The GPGPU core 1562 may include SIMD logic capable of executing a single instruction on multiple sets of data. The GPGPU core 1562 can physically execute SIMD4, SIMD8, and SIMD16 instructions, and logically execute SIMD1, SIMD2, and SIMD32 instructions. The SIMD instructions for the GPGPU core can be generated by the shader compiler at compile time, or automatically generated when executing programs written and compiled for Single Program Multiple Data (SPMD) or SIMT architectures. Multiple threads of a program can be configured for a SIMT execution model that can be executed via a single SIMD instruction. For example, eight SIMT threads performing the same or similar operations can be executed in parallel via a single SIMD8 logic unit.

[0160] The memory and cache interconnect 1568 may include an interconnect network that connects each functional unit of the graphics multiprocessor 1534 to the register file 1558 and shared memory 1570. The memory and cache interconnect 1568 may be a crossbar interconnect that allows the load / store unit 1566 to perform load and store operations between the shared memory 1570 and the register file 1558. The register file 1558 may operate at the same frequency as the GPGPU core 1562, thus data transfer between the GPGPU core 1562 and the register file 1558 can have very low latency. The shared memory 1570 can be used to implement communication between threads executing on functional units within the graphics multiprocessor 1534. The cache memory 1572 can be used as a data cache, for example, for caching texture data transferred between functional units and texture units 1536. The shared memory 1570 can also be used as a program-managed cache. In addition to automatically caching the data stored in cache memory 1572, threads executing on GPGPU core 1562 can also programmatically store data in shared memory.

[0161] The parallel processor or GPGPU described herein can be communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., high-speed interconnects, such as, but not limited to, PCIe or NVLink). A System-on-a-Chip (SoC) may include the parallel processor or GPGPU described herein, which executes on the SoC. The GPU may be integrated as a core on a package or chip and communicatively coupled to the core via an internal processor bus / interconnect within the package or chip. Regardless of the GPU's connection method, the processor core can assign work to the GPU in the form of a sequence of commands / instructions contained in a job descriptor. The GPU can then use dedicated circuitry / logic to efficiently process these commands / instructions to perform any of the operations described above or elsewhere herein.

[0162] In at least one embodiment, the graphics multiprocessor 1534 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0163] Figure 16A processor 1600 according to at least one embodiment is illustrated. The processor 1600 may include a hybrid architecture processor (e.g., Lunar Lake or Meteor Lake) from Intel Corporation, Santa Clara, California, or other processors sharing at least some of the components described herein. The processor 1600 may include one or more central processing units (CPU 1602), one or more graphics processing units (GPU 1606), and / or one or more neural processing units (NPU 1608), which may be, for example, dedicated AI accelerators for offloading artificial intelligence (AI) workloads from the CPU 1602 and GPU 1606. The processor 1600 may use instructions that, if executed, cause the processor 1600 and / or any of its components to perform some or all of the processes and techniques described elsewhere herein. The processor 1600 may include any number of memory and cache units 1610 for facilitating processing between different components of the processor 1600. The memory and cache 1610 on processor 1600 may include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. Regarding processor 1600 and any components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to processor 1600. The results of APIs may be stored in storage devices internal or external to processor 1600, including registers, DRAM, flash memory, SRAM, cache, or other memory. One or more APIs described herein may include calls.

[0164] The processor 1600 may include the computing engine as the CPU 1602, and may include any number of cores, such as, but not limited to, up to 16 cores / 22 threads. The cores in the CPU 1602 may include P-cores (performance), E-cores (high efficiency), and LP-E cores (low-power, high-efficiency). Performance cores can be used for low-latency, single-threaded, compute-intensive workloads, while high-efficiency cores can be used for multi-threaded, less compute-intensive workloads. Low-power, high-efficiency cores can be used for scalable multi-threaded execution and offloading of background instructions. P-cores can be used for single-threaded and limited-threaded execution, while E-cores and LP-E cores are used for multi-threaded throughput and power efficiency.

[0165] The GPU 1606 can include any number of graphics engines, such as, but not limited to, the Intel® Arc™ Graphics Engine (Xe LPG) with 8 Xe cores (up to 128 execution units or EUs). Figure 16 As shown, GPU 1606 may include a vector engine 1610 and a matrix engine 1612, which, for example, can execute FP, INT, and matrix operation instructions simultaneously, individually, or in batches. GPU 1606 may include a load / store unit 1614, as well as other memories, such as, but not limited to, an instruction cache (I$) 1616 and an L1 cache / subsystem local memory (SLM) 1618, which may, for example, store instructions for performing any of the operations described above or elsewhere herein.

[0166] The NPU 1604 may include one or more Intel® AI Boost built-in Neural Processing Units (NPUs). The NPU 1604 can be enumerated as an integrated PCIe device to the host processor. The NPU 1604 may include one or more (e.g., two) Neural Computation Engine (NCE) tiles 1630. Each tile may be configured with any combination of, but not limited to, the following: (e.g., 2000) Multiply-Accumulate (MAC) engines 1634, a post-processing engine (not shown), an AI DSP processor (not shown), and memory per tile (2MB dedicated SRAM), such as... Figure 16 As shown. For general computing needs, the neural computing engine 1630 may include a disturbance pipeline 1632, an activation function (AF) 1636, a data transformation 1638, a load / store 1640, and a streaming hybrid architecture vector engine (SHAVE) 1628 for high-performance parallel computing, which may include a DMA (Direct Memory Access) engine 1624 for transporting data between system memory DRAM (Dynamic Random Access Memory) 1626 and a software-managed cache. The built-in device MMU (Memory Management Unit) 1622, plus the IOMMU (Input-Output Memory Management Unit) (not shown), can support multiple concurrent hardware contexts and provide secure isolation between execution contexts according to the MCDM (Microsoft Computing Driver Model) architecture. The processor 1600 may also include a media unit (not shown), which may be included on or separate from the XCD or other components of the processor 1600 to enable video playback and video processing of compressed or uncompressed data, such as using HEVC, AV1, VP9, ​​and AVC hardware-accelerated decoding support and HEVC, VP9, ​​and AVC hardware-accelerated encoding support.

[0167] The Intel® Thread Director (which includes firmware built into the processor 1600) prioritizes and manages workload distribution, sending instructions to optimized cores. For example, the Thread Director can tie P-cores, E-cores, and / or LP-E-cores (as described above) together with instruction scheduling capabilities and the ability to send less demanding instructions to E-cores or LP-E-cores. Intel® Deep Learning Boost (Intel® DL Boost, not shown) provides built-in AI acceleration for training and inference workloads and may include VNNI (for CPU) and DP4a (for GPU) instruction set support. This instruction set can be optimized using the OpenVINO™ toolkit and oneAPI to accelerate INT8 inference. For example, a software stack as described elsewhere in this document can be used to implement AI inference using the OpenVINO™ toolkit. The processor 1600 can be configured to execute applications, such as, but not limited to, CUDA programs.

[0168] In at least one embodiment, the processor 1600 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0169] Processor 1600 may alternatively include a processor based on Qualcomm's AIEngine Direct architecture from Santa Clara, California, or other processors sharing at least some of the components described herein. It may include any number of NPUs, GPUs, CPUs, and other associated components, such as, but not limited to, an NPU 1604 as a Hexagon NPU, a GPU 1606 as an Adreno GPU, a CPU 1602 as a Kryo or Qualcomm Oryon CPU, and a Qualcomm Sensing Hub (not shown) and a memory subsystem 1610 in any combination. Hexagon NPU 1604 may include power rails, micro-tile inference units, hardware acceleration units, tensor units, scalar units, and vector units (all not shown), which may have dedicated or shared memory (e.g., cache or memory, such as HBM3) for storing instructions for performing any of the operations described above or elsewhere herein. The Adreno GPU 1606 can provide graphics and parallel processing for AI, in formats including but not limited to 32-bit floating-point (FP32), 16-bit floating-point (FP16), and 8-bit integer (INT8). The Kryo or Qualcomm Oryon CPU 1602 can execute AI workloads and handle the contextualization of ubiquitous generative AI applications. The CPU 1602 may also include an instruction fetch unit, a renaming and deprecation unit, a memory management unit, a vector execution unit, an integer execution unit, and a load and store unit for processing and instruction management. Regarding the processor 1600 and any of its components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by the instruction fetch unit, decoded by the processor decoder or equivalent, scheduled (e.g., sequentially or out of order) for execution by the scheduler or equivalent, executed by the execution logic or equivalent, reordered, and then deprecated by the renaming and deprecation unit. The API (and / or compiled instructions including the API) can be stored in any storage device (e.g., cache and / or memory) inside or outside the processor 1600. An arbitrary number of CPU cores 1602 can be included in an arbitrary number of CPU clusters, which can be coupled to memory and / or cache, such as, but not limited to, a shared L2 cache. Memory can be separate or shared; for example, the CPU clusters of CPU cores 1602 can be coupled to a memory subsystem 1610, which can include structures capable of reading and writing to memory (e.g., DRAM), system-level caches, and an arbitrary number of memory management units.The Qualcomm sensing hub (not shown) includes a miniature NPU, power rails, and conventional sensors (such as gyroscopes, accelerometers, or even barometers) that support voice and data streaming. The memory subsystem 1610 may include memory and cache on the processor 1600, which may include L1 or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination, for example, for storing information and / or instructions for performing any of the operations described above or elsewhere herein. All or part of the memory and / or cache in the memory subsystem 1610 may be shared or used individually by any component or combination of components on the processor 1600 (e.g., GPU 1606, NPU 1604, and CPU 1602).

[0170] The Qualcomm AI Engine 1600 can be programmed and controlled using a software stack to perform some or all of the operations described herein, including, for example, versions of the Qualcomm® Neural Processing SDK for inference on Android, Linux, and Windows. Developer libraries and services support programming languages, virtual platforms, and compilers. At lower levels of the software stack, system software includes a basic real-time operating system (RTOS), system interfaces, and drivers. The software stack supports various operating systems, including Android, Windows, Linux, and QNX, as well as deployment and monitoring infrastructures such as Prometheus, Kubernetes, and Docker. For direct cross-platform access to the GPU 1606, OpenCL and DirectML are supported. For the CPU 1602, LLVM compiler infrastructure optimizations enable accelerated and efficient AI inference. Regarding the Qualcomm AI Engine 1600 and any components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic or equivalents, decoded by processor decoders or equivalents, scheduled (e.g., sequentially or out of order) by a scheduler or equivalent for execution, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. The APIs (and / or compiled instructions including the APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to the Qualcomm AI Engine 1600. The results of the APIs may be stored in storage devices internal or external to the Qualcomm AI Engine 1600, including registers, DRAM, flash memory, SRAM, cache, or other memory.

[0171] In at least one embodiment, the processor 1600 or the Qualcomm AI engine 1600 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0172] Figure 17A A processor 1700 according to at least one embodiment is illustrated. The processor 1700 may include a Scalable family of processors from Intel Corporation, Santa Clara, California, or other processors that share at least some of the components described herein. The processor 1700 may include one or more cores 1712(1)-1712(N) capable of performing the operations described elsewhere herein, where N is any integer greater than 1. Cores 1712(1)-1712(N) may be interconnected using ring and / or mesh interconnects. Utilizing a mesh interconnect architecture, arrays of vertical and horizontal communication paths may allow traversal from one core to another 1712(1)-1712(N) via the shortest path (jumping to the correct row along the vertical path and to the correct column along the horizontal path). For the mesh interconnect, a die may accommodate cores 1712(1)-1712(N) and may include a Converging Mesh Stop Point (CMS) grid that may be associated with cores 1712(1)-1712(N) (e.g., 1:1). Each core may be associated with a low-level cache (LLC) slice 1714(1)-1714(N), or cores 1712(1)-1712(N) may share a cache, such as a low-level cache. LLC 1714(1)-1714(N) may be inclusive or non-inclusive (having blocks not present in the high-level cache) by merging blocks from a higher-level cache (e.g., L2 cache). Each core and LLC slice may include a caching and home agent (CHA) (not shown) that maintains cache coherency by providing resource scalability via mesh interconnect, thereby enabling cache coherency features of Intel® HyperPath Interconnect (Intel® UPI 1716). UPI 1716 provides a coherent interconnect for scalable systems and allows multiple processors to share a single shared address space via links, such as, but not limited to, two or three UPI links per processor.

[0173] The processor 1700 may also include a system agent 1710, which may house and / or perform various functions, such as, but not limited to, memory management, display functions, and / or input / output (I / O) functions. For example, the processor 1700 may include one or more integrated memory controllers (IMCs) 1708. The IMC 1708 may control and manage memory, such as, but not limited to, different memory types, such as DDR RAM, such as DDR4, or other memory described elsewhere herein. The system agent 1710 may include a display controller (not shown) for supporting one or more displays. The system agent 1710 may also integrate a PCIe 1704 (e.g., up to 20 PCIe lanes), which may, for example, be connected to an external dedicated graphics connector via a DMI bus (e.g., Intel's DMI 3.0 bus) 1706. The system agent 1710 may include an image processing unit (IPU) (not shown) that integrates an on-die image signal processor (ISP). Architecture 1702 provides scalability to connect to other nodes, such as processors, like processor 1700, and can be used, for example, with Cornelis Networks, an element of the Intel® Scalable System Framework, which provides performance for high-performance computing (HPC) workloads and the ability to scale to tens of thousands of nodes.

[0174] Figure 17B Components within a core 1712 according to at least one embodiment are illustrated. The core 1712 may include a front-end 1718, a back-end or execution engine 1732, and a memory subsystem 1742. The front-end 1718 may provide operations (e.g., operations described elsewhere herein) to the execution engine 1732 by decoding instructions stored in memory. For example, the front-end 1718 may include micro-operation (µOps) cache paths and / or traditional paths, and a branch prediction unit 1721 that can determine path instructions. A traditional path for instructions may include fetching variable-length (e.g., x86) instructions from an L1 instruction cache 1720 and instruction fetch and pre-decode 1722, queuing these instructions into an instruction queue 1724, and decoding the instructions into µOps that can be provided to an allocation queue 1728 using a decoder 1726. Alternatively, the µOps cache path may include a cache that includes decoded µOps (µOps 1730) that can be sent to the allocation queue 1728. The allocation queue 1728 can act as an interface between the front end 1718 and the execution engine 1732, and can provide instructions to the execution engine 1732. For example, one or more APIs described herein can be compiled into instructions that can be stored, processed, and executed by the front end 1718 and the execution engine 1732, and stored in the memory subsystem 1742.

[0175] Execution engine 1732 can receive micro-operations into reordering buffer 1734, which can register allocations, renames, and decommissions of µOPs. µOPs can be sent from the reordering buffer to scheduler 1736, which can be connected to one or more different execution units 1738, which can be connected to address generation units (AGUs) 1740. Execution units 1738 can perform operations such as basic arithmetic logic unit (ALU) operations, multiplication, division, and / or more complex operations, such as, but not limited to, various vector operations. Scheduler 1736 can manage the queuing of µOPs for one or more execution units 1738 based on, for example, the operations that need to be performed.

[0176] The memory subsystem 1742 can handle load and store requests as well as sorting operations. For example, µOPs may be associated with memory accesses (e.g., load and store), and these µOPs can be sent through dedicated scheduler ports that can perform these memory operations. For example, store and load operations can be sent to load and store buffers 1744. The memory subsystem 1742 may also include shared or separate L1 data and instruction caches 1746, and an L2 cache 1748 that can be used and shared by the L1 data and instruction caches 1746. (As described above regarding...) Figure 17A Each core 1712 can be connected to a slice of a level 3 cache (e.g., LLC1714), which can be shared by all cores 1712.

[0177] In at least one embodiment, processor 1700 may include one or more circuits configured to execute an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0178] Figure 18An AI accelerator 1800 according to at least one embodiment is illustrated. The processor 1800 may include a processor with an AI accelerator architecture manufactured by Intel Corporation, Santa Clara, California, or other processors sharing at least some of the components described herein. The AI ​​accelerator 1800 may use instructions that, if executed by the AI ​​accelerator 1800, cause the AI ​​accelerator 1800 to perform some or all of the processes and techniques described elsewhere herein. For example, with respect to the AI ​​accelerator 1800 and any components described above or elsewhere herein, one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic or equivalents, decoded by processor decoders or equivalents, scheduled (e.g., sequentially or out of order) by a scheduler or equivalent for execution, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. The APIs (and / or compiled instructions including the APIs) may be stored in any storage device internal or external to the AI ​​accelerator 1800 (e.g., in cache and / or memory). The results of the API can be stored in internal or external storage devices of the AI ​​accelerator 1800, including registers, DRAM, flash memory, SRAM, cache, or other memory. The AI ​​accelerator 1800 may include one or more compute dies, which may include homogeneous or heterogeneous processors. The compute dies may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), or a combination of both.

[0179] In at least one embodiment, the computational die may include a computational engine for performing AI computations. In at least one embodiment, the computational die of the AI ​​accelerator 1800 may be split into any number (e.g., four) clusters, which may be referred to as DCORE (Deep Learning Core) 1806, and include any number of matrix multiplication engines (MME) 1808, tensor processor cores (TPC) 1810, memory management units 1812, and L2 caches 1814 in any combination. The MME 1808 may perform operations using matrix multiplication, such as fully connected layers, convolutions, and batch general matrix multiplication (GEMM). The MME 1808 may be equipped with a multiplication-accumulation unit (MAC) (not shown), which may perform general matrix multiplication (GEMM) operations, such as, but not limited to, AxB multiplication, which involves generating a tensor C [NxM] from two input tensors A [NxK] and B [KxN]. The MME 1808 may be programmed with array dimensions, positions, data types, and various operands. The MME 1808 can retrieve tensors A and B from memory and pull them into its streaming buffer for matrix multiplication to be performed in parallel by the MAC. After completion, the MME 1808 can push tensor C back to memory. The TPC 1810 may include any number of scalar units for performing scalar operations, any number of vector units for performing vector operations, any number of register files or local memory units (e.g., vector local memory), and load and store components for instructions, which may be coupled to memory or caches (e.g., HBM, L3 cache, and / or L2 cache) (all not shown). The TPC can support different types of parallel processing, such as Very Long Instruction Word (VLIW) Single Instruction Multiple Data (SIMD) data types such as, but not limited to, FP32, BF16, FP16, and FP8 (both E4M3 and E5M2), UINT32, INT32, UINT16, INT16, UINT8, and INT8 data types. Any number of computational dies can be interconnected. Interconnects that can connect computing dies can be via intermediate bridges, which are transparent to software, for example.

[0180] The memory on the AI ​​accelerator 1800 may include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. The memory and / or cache system may be unified or separate. The compute die of the AI ​​accelerator 1800 may include on-chip memory comprising one or more levels of cache (e.g., two levels). On-chip SRAM or other memory described elsewhere herein may be used as a unified last-level cache (L3) or split into multiple slices of L2 cache accessible to the MME 1808 and TPC 1810 groups. Using on-chip memory as an L2 or L3 cache can be fully configurable by software, which can dynamically determine its optimal cache allocation based on I / O tensors. The AI ​​accelerator 1800 may include one or more memory management units (MMUs) 1822 for managing memory, such as allowing the AI ​​accelerator 1800 memory subsystem to operate in virtual space when accessing VRAM.

[0181] AI accelerator 1800 may include a communication port (e.g., a PCIe Gen5 x16 port) 1802 for communicating with a host and scheduling and synchronization unit 1804. AI accelerator 1800 may include a media unit 1816, which may include any number or combination of media decoder engines (DECs) 1820 and rotation engines (ROTs) 1818. AI accelerator 1800 may include a network unit 1824, which may include any number or combination of network ports 1826 and an accompanying RDMA engine 1828, L2 cache, and memory (e.g., HBM2e or HBM3) stack. AI accelerator 1800 may include a programmable control path entity (not shown) for managing the parallel and efficient execution of the various engines. The control path may include a submission queue (SQ) that can be issued by the runtime system, a completion queue (CQ) that can be used for job completion reporting, a programmable scheduling mechanism that can be used for instruction scheduling, a programmable hardware synchronization mechanism or "synchronization manager (SM)" that can be used for hardware synchronization, and a programmable interrupt service mechanism or "interrupt manager (INTR)" that can pass asynchronous events to drivers.

[0182] The AI ​​Accelerator 1800 may include media decoding units supporting video formats such as, but not limited to, HEVC, Progressive H.264, SVC base layer, MVC, VP9, ​​JPEG, and Progressive JPEG. The AI ​​Accelerator 1800 may support post-processing of the decoded media stream, such as, but not limited to, image downsizing (image resizing), vertical and horizontal scaling at different scaling ratios, image enlargement, image cropping, bilinear scaling, and Lancos scaling. The AI ​​Accelerator 1800 may implement two post-processing channels per decoder unit, one for scalar (up and down) and the other solely for outputting the original image. The AI ​​Accelerator 1800 may include a hardware rotation engine that performs the following transformations on the input image: 2D rotation, 3D rotation, projection, image warping and de-warping, resampling of the input data at user-defined coordinates, and rescaling.

[0183] The AI ​​accelerator 1800's RDMA 1828 based on converged Ethernet enables scaling from a single node (i.e., from a single AI accelerator 1800 to hundreds or thousands of nodes or AI accelerators 1800). The network subsystem 1824 may include the Intel® Gaudi® Communications Library (IGCL), a master controller coordinating data movement, and a programmable scheduling mechanism that enables smooth engine activation while maintaining instruction dependencies. The accelerator network subsystem may include a Gigabit Ethernet NIC port 1826, a Layer 2 MAC (not shown), and the RDMA engine 1828. The AI ​​accelerator 1800 may include an aggregation engine for performing summation activities. All engines in the processor 1800 can run in parallel; for example, the MME 1808, TPC 1810, and NIC 1826 can all operate simultaneously. Dependencies may exist between operations running on different engines; for example, the output of one engine may be used as the input of another engine, and / or the MME, TPC, and NIC may be scheduled to run in parallel. Once one engine has finished its operation, another engine can be scheduled to begin working on the next operation (executed immediately once its input is ready).

[0184] The AI ​​accelerator 1800 can be operated and controlled using a software layer 1828, which may include low-level components such as, but not limited to, a graph compiler, an automatic kernel fusionist and pre-compiled kernel libraries, and integrations with the AI ​​ecosystem such as, but not limited to, PyTorch, DeepSpeed, Hugging Face, vLLM, Ray, etc., or as described elsewhere in this document regarding software and programming platforms. The software layer 1828 may include implementations of algorithms such as, but not limited to, PagedAttention, FlashAttention, etc. The software layer 1828 can generate optimized binary code that implements a given model topology, such as, but not limited to, performing operator fusion, data layout management, parallelization, pipeline and memory management, and graph-level optimization.

[0185] In at least one embodiment, the AI ​​accelerator 1800 may include one or more circuits configured to execute an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. The one or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0186] This paper describes a neuromorphic computing system employing a multi-core architecture, where each core houses computing elements including neurons, synapses with on-chip learning capabilities, and local memory for storing synaptic weights and routing tables. Figure 19A simplified block diagram 1900 illustrating at least a portion of such a neuromorphic computing device 1905 according to at least one embodiment. The neuromorphic computing device 1905 may include a neuromorphic processor from Intel Corporation, Santa Clara, California, or other processors that include at least a portion of the components described herein. As shown in this example, the device 1905 may be equipped with a network 1910 consisting of multiple neural network cores interconnected by a network on the device, thereby potentially defining multiple distinct connections between the cores. For example, the device 1905 may provide a network 1910 of spiking neural network cores, each core communicating via short packet pulse messages sent from one core to another through network channels. Each core (e.g., 1915) may have processing and memory resources, as well as logic, for implementing a number of primitive nonlinear time computation elements, such as, but not limited to, multiple (e.g., more than 1000) distinct artificial neurons (referred to herein as “neurons”). For example, each core may be able to implement multiple neurons concurrently, allowing the neuromorphic core to implement many, many neurons using the device 1905. With respect to the neuromorphic computing device 1905 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) by scheduler or equivalents for execution, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to the neuromorphic computing device 1905. The results of the APIs may be stored in storage devices internal or external to the neuromorphic computing device 1905, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0187] continue Figure 19For example, the neuromorphic computing device 1905 may also include a processor 1920 and a system memory 1925 for implementing one or more components to manage and provide the functionality of the neuromorphic computing device 1905. For instance, a system manager 1930 may be provided to manage the global attributes and operations of the neuromorphic computing device 1905 (e.g., attributes affecting the core network 1910, multiple cores in network 1910, interconnection of the neuromorphic computing device 1905 with other devices, managing access to the global system memory 1925, and other potential examples). In one example, the system manager 1930 may manage the definition and configuration of specific routing tables for individual routers in network 1910, orchestration of network definitions and attributes to be applied to network 1910 (e.g., weights, attenuation rates, etc.), core synchronization and time multiplexing management, routing input to the appropriate core, and other potential functions.

[0188] As another example, the neuromorphic computing device 1905 may also include a programming interface 1935 through which a user or system can specify the neural network definition to be applied (e.g., via routing tables and individual neuron attributes), implemented by the neuromorphic core grid 1910. A software-based programming tool may be provided with or separately from the neuromorphic computing device 1905, through which a user can provide a definition for a specific neural network to be implemented using the neuromorphic core network 1910. The programming interface 1935 can receive input from the programmer, then generate the corresponding routing table and populate the specified parameters into the local memory of each neuromorphic core (e.g., 1915) to implement the corresponding custom artificial neural network implemented by the neuromorphic core 1915.

[0189] In certain situations, the neuromorphic computing device 1905 can advantageously engage and interoperate with other devices, including general-purpose computing devices, to enable specific applications and use cases. Therefore, in some cases, external interface logic 1940 may be provided to communicate with one or more other devices (e.g., via one or more defined communication protocols). External interface 1940 may be used to accept input data from another device or an external memory controller used as an input data source. External interface 1940 may additionally or alternatively be used to allow the results or outputs of computations performed using the neural network implemented with the neuromorphic computing device 1905 to be provided to another device (e.g., another general-purpose processor implementing machine learning algorithms) to enable additional applications and enhancements, among other examples.

[0190] like Figure 19The diagram illustrates a network 1910 of multiple neural network cores interconnected by a network on the device, showing a portion of a network structure interconnecting multiple neuromorphic cores (e.g., 1915a-d). For example, several neuromorphic cores (e.g., 1915a-d) can be provided in a mesh, each core interconnected via a network including multiple routers (e.g., 1950). In one implementation, each neuromorphic core (e.g., 1915a-d) can be connected to a single router (e.g., 1950) in the router array, and each router can be connected to at least one other router (e.g., [other router]). Figure 19 (As shown at 1910 in the diagram). As an example, in one particular implementation, four neuromorphic cores (e.g., 1915a-d) can be connected to a single router (e.g., 1950), and each router 1950 can be connected to two or more other routers to form a multi-core mesh, thereby allowing each neuromorphic core to interconnect with every other neuromorphic core in the neuromorphic computing device 1905. Furthermore, since each neuromorphic core can be configured to implement multiple different neurons, the router network of the neuromorphic computing device 1905 can similarly implement connections or artificial synapses (or simply "synapses") defined between any two of the potentially many (e.g., 30,000+) neurons defined by the network definition using the neuromorphic cores 1910 provided in the neuromorphic computing device 1905.

[0191] Figure 19A block diagram of the internal components of an example implementation of the neuromorphic core 1915 is shown. In one example, a single neuromorphic core may implement a number of neurons (e.g., 1024), which share the architectural resources of the neuromorphic core 1915 in a time-division multiplexing manner. In one example, each neuromorphic core 1915 may include a processor block 1955 capable of executing arithmetic functions and routing related to the implementation of the digitally implemented artificial neurons, such as, but not limited to, those explained herein. Each neuromorphic core 1915 may also provide local memory in which routing tables of the neural network can be stored and accessed, accumulated potentials of each cell body of each neuron implemented using core 1915 can be tracked, parameters of each neuron implemented by core 1915 can be recorded, and other data and usage can be recorded. Components or architectural resources of the neuromorphic core 1915 may also include: an input interface 1965 for receiving input spike messages generated by other neurons on other neuromorphic cores; and an output interface 1970 for sending spike messages to other neuromorphic cores via a mesh network 1910. In some instances, the routing logic of the neuromorphic core 1915 can be implemented at least partially using the output interface 1970. Furthermore, in some cases, the core (e.g., 1915) can implement multiple neurons within an example SNN, and some of these neurons can be interconnected. In this case, spiking messages sent between neurons hosted on the core 1915 can forgo communication via the routing structure of the neuromorphic computing device 1905 and can be managed locally within the specific neuromorphic core 1915.

[0192] Each neuromorphic core may also include logic for implementing artificial dendrites 1980 and artificial cell bodies 1985 (hereinafter referred to as “dendrites” and “cell bodies”, respectively) for each neuron 1975. Dendrite 1980 may be a hardware-implemented process for receiving impulses from network 1910. Cell body 1985 may be a hardware-implemented process for receiving the current time-accumulated neurotransmitter mass of each dendrite and evolving the potential states of each dendrite and cell body to generate outgoing impulse messages at appropriate times. Dendrite 1980 may be defined for each connection receiving input from another source (e.g., another neuron). In one implementation, the dendritic process 1980 may receive and process the impulse message as it arrives serially from network 1910 in a time-division multiplexed manner. With the reception of impulses, neuronal activation (tracked using cell body 1985 (and local memory 1960)) may increase. When the activation of a neuron exceeds a threshold set for neuron 1975, neuron 1975 generates a spike message, which is propagated via output interface 1970 to a fixed set of fan-out neurons. The network distributes the spike messages to all destination neurons, which in turn can update their activation in a transient, time-dependent manner in response. This can lead to some of the destination neurons also exceeding their corresponding thresholds and triggering further spike messages, just as in real biological neural networks.

[0193] As described above, the neuromorphic computing device 1905 can reliably implement spiking-based neural computing models. Such models are also known as spiking neural networks (SNNs). In addition to neuronal and synaptic states, SNNs incorporate temporal concepts. For example, in SNNs, communication occurs via event-driven action potentials or spiking, which convey no explicit information other than the spiking time and the implicit source and destination neuron pairs corresponding to the spiking transmission. The computation of the result of a dynamic nonlinear integral as a weighted spiking input occurs in each neuron. In some implementations, recurrent and dynamic feedback can be incorporated into the SNN computation model. Furthermore, various network connectivity models can be employed to model a wide range of real-world networks or relationships, including fully connected (all-to-all) networks, feedforward trees, completely random projections, "small-world" networks, and other examples. Isomorphic two-dimensional networks at the neuromorphic core (e.g., but not limited to...) Figure 19The network shown in the example can advantageously support all these network models. Since some or all of the cores of the neuromorphic computing device 1905 can be connected, some or all of the neurons defined in a core can also be fully connected via a certain number of router hops. The neuromorphic computing device 1905 can also include fully configurable routing tables for defining various neural networks by allowing neurons in each core to distribute their spurs to any number of cores in the grid 1910 to achieve a completely arbitrary connection graph.

[0194] In improved implementations of systems capable of supporting SNNs, for example, but not limited to... Figure 19 The example illustrates a very large-scale integrated circuit (VLSI) hardware device that can provide high-speed, reliable circuitry to implement SNNs (Spiritual Neural Networks) to model the information processing algorithms used by the brain, but in a more programmable way. For instance, while a biological brain can only perform a specific set of defined behaviors (a consequence of years of development), a neuromorphic processor device can provide the ability to rapidly reprogram all neural parameters. Therefore, a single neuromorphic processor can be used to implement a wider range of behaviors than a single slice of biological brain tissue. This distinction can be achieved by employing neuromorphic processors with neuromorphic designs that are distinctly different from those found in natural neural circuitry.

[0195] As an example, a neuromorphic processor can implement a spontaneous neural network (SNN) using time-multiplexed computation in both a spiking communication network and the neuronal mechanism of the neuromorphic computing device 1905. Therefore, the physical circuitry of the neuromorphic computing device 1905 can be shared by many neurons to achieve a higher neuron density. Through time multiplexing, the network can connect N cores with a total wiring length of O(N), while the length of discrete point-to-point wiring will be extended to O(N). 2 This significantly reduces wiring resources to accommodate planar and non-plastic VLSI routing techniques, among other examples. In the neuromorphic core, time multiplexing can be implemented through dense memory allocation, for example, using static random access memory (SRAM) with a shared bus, address decoding logic, and other multiplexed logic elements. The state of each neuron can be stored in the processor's memory, where data describing the state of each neuron includes the state of the collective synapse of each neuron, all currents and voltages on its membrane, and other example information (e.g., but not limited to configuration and other information).

[0196] Neuromorphic processors can be implemented in a “digital” manner, unlike other processors that employ more “analog” or “isomorphic” neuromorphic approaches. For example, a digital implementation can use digital adder and multiplier circuitry to integrate synaptic currents, in contrast to an analog isomorphic neuromorphic approach that accumulates charge on capacitors in a manner similar to how neurons accumulate synaptic charge on their lipid membranes. For instance, the accumulated synaptic charge for each neuron can be stored in the local memory of the corresponding core. Furthermore, at the architectural level of an example digital neuromorphic processor, reliable and deterministic operation can be achieved through time synchronization across the core network, ensuring that any two executions of the design, given the same initial conditions and configuration, will produce the same results. Asynchronicity can be reserved at the circuit level to allow individual cores to operate as quickly and freely as possible while maintaining determinism at the system level. Therefore, in neural computing, the concept of time as a time variable can be abstracted away from the “wall clock” time used by the hardware to perform computations. Thus, in some implementations, a time synchronization mechanism can be provided that globally synchronizes the neuromorphic cores at discrete time intervals. The synchronization mechanism allows neural computation to be completed at the fastest speed allowed by the circuit, and there is a difference between the runtime and the biological time for modeling neuromorphic systems.

[0197] In operation, the neuromorphic computing device 1905 can start in an idle state when all neuromorphic cores are inactive. As each core asynchronously loops through its neurons, it generates impulse messages, which are routed by the mesh interconnect to the appropriate destination core containing all destination neurons. The implementation of multiple neurons on a single neuromorphic core can be time-multiplexed, and time steps can be defined, where all impulses involving multiple neurons can be processed and considered using the shared resources of the respective cores. When each core completes its service to its neurons within the corresponding time step, in some implementations, the core can communicate with neighboring cores using synchronization messages (e.g., using a handshake) to refresh the mesh of all transmitted impulse messages, allowing the core to safely determine that all impulses have been serviced within a given time step. At this point, all cores can be considered synchronized, allowing them to advance their time steps and return to their initial state to begin the next time step.

[0198] Given this context, as described above, a device (e.g., 1905) can be provided to realize an interconnected neuromorphic core grid 1910, wherein core 1915 can realize multiple artificial neurons capable of interconnecting to realize an SNN. Each neuromorphic core (e.g., 1915) can provide two loosely coupled asynchronous processes: an input dendrite process (e.g., 1980) that receives impulses from network 1910 and applies them to the appropriate destination dendritic compartment at an appropriate future time; and an output cell body process (e.g., 1985) that receives the current-time accumulated neurotransmitter mass of each dendritic compartment and evolves the membrane potential state of each dendrite and cell body to generate an outgoing impulse message at an appropriate time (e.g., when the threshold potential of the cell body is reached). It should be noted that, from a biological perspective, the names of dendrites and cell bodies used herein are only approximate to the function of these features and should not be interpreted too literally.

[0199] In at least one embodiment, the neuromorphic computing device 1905 may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. The one or more circuits may be software-configured to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0200] Figure 20 This is a block diagram of an embodiment of a multi-node network capable of enabling remote memory computing according to any embodiment. System 2000 may represent the node network described herein, for example, the node network may be used to perform some or all of the operations described herein. System 2000 may represent a data center. System 2000 may represent a server farm. System 2000 may represent a data cloud or processing cloud. System 2000 may represent a supercomputer. System 2000 may include tens, hundreds or thousands of nodes. The nodes of System 2000 may include processors, such as, but not limited to, a central processing unit (CPU), a graphics processing unit (GPU), or any combination of processors described herein, such as, but not limited to, processors described herein. Figure 14-26BOther processors in the system. For any processor in System 2000 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) inside or outside the processor or node. The results of APIs may be stored in storage devices inside or outside the processor or node, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents. System 2000 may include more than nine thousand nodes, each node comprising two Intel Xeon Max processors, six Intel Max series GPUs, and a unified memory architecture, such as, but not limited to, the architecture used in Intel's Aurora supercomputer in Santa Clara, California, or other supercomputers that share at least some of the components described herein.

[0201] One or more clients 2002 send requests to system 2000 via network 2004. Network 2004 represents one or more local area networks, wide area networks, or a combination of both. Clients 2002 can be human or machine clients that generate requests for operations to be performed by system 2000. System 2000 executes the application or data calculation instructions requested by client 2002.

[0202] System 2000 may include one or more racks, which represent structural and interconnect resources for housing and interconnecting multiple computing nodes. Rack 2010 may include multiple nodes 2030. Rack 2010 may carry multiple blade assemblies 2020(0)-2020(N-1), where N is an integer greater than or equal to 2. Carrying may refer to providing power, structural or mechanical support, and interconnection. Blades 2020(0)-2020(N-1) may refer to computing resources on a printed circuit board (PCB), where the PCB houses the hardware components of one or more nodes 2030. Blades 2020(0)-2020(N-1) may include or exclude a chassis, enclosure, or other “box” besides those provided by rack 2010. Blades 2020(0)-2020(N-1) may include an enclosure with exposed connectors for connection to rack 2010. System 2000 may or may not include rack 2010, and each blade (e.g., 2020(0)) may include a chassis or housing that may be stacked or otherwise closely proximate with other blades and allow nodes 2030 to interconnect. System 2000 may include 10,624 compute blades, comprising 63,744 Intel Max series GPUs and 21,248 Intel Xeon Max CPUs on 166 racks.

[0203] System 2000 may include architecture 2070, which represents one or more interconnectors of nodes 2030. Architecture 2070 may include multiple switches 2072 or routers or other hardware for routing signals between nodes 2030. Furthermore, architecture 2070 may couple system 2000 to network 2004 for access by client 2002. In addition to routing devices, architecture 2070 may also be considered to include cables or ports or other hardware devices for coupling nodes 2030 together. Architecture 2070 may have one or more associated protocols for managing signal routing through system 2000. One or more protocols are at least partially dependent on the hardware devices used in system 2000.

[0204] As shown in the figure, rack 2010 may include N blades (e.g., 2020(0)-2020(N-1)). In addition to rack 2010, system 2000 may also include rack 2050. As shown in the figure, rack 2050 may include M blades (e.g., 2060(0)-2060(M-1)). M is not necessarily the same as N; therefore, it is understood that various different hardware device components may be used and coupled together into system 2000 via structure 2070. Blades 2060(0)-2060(M-1) may be the same as or similar to blades 2020(0)-2020(N-1). Node 2030 may be any type of node described herein and is not necessarily of the same type. System 2000 is not limited to homogeneous or non-homogeneous.

[0205] The nodes in blade 2020(0) are shown in detail. However, other nodes in system 2000 may be the same or similar. At least some nodes 2030 may be compute nodes, having processors 2032 and memory 2040. A compute node is a node having processing resources (e.g., one or more processors) that executes an operating system and can receive and process one or more instructions. At least some nodes 2030 may include storage server nodes, which have servers as processing resources 2032 and memory 2040. A storage server is a node having more storage resources than a compute node, and instead of having processors for executing instructions, a storage server includes processing resources for managing access to storage nodes within the storage server.

[0206] Node 2030 may include interface controller 2034, which may represent logic for controlling node 2030's access to structure 2070. The logic may include hardware resources for interconnecting to physical interconnect hardware. The logic may include software or firmware logic for managing the interconnect. Interface controller 2034 may include a host structure interface, which may include a structure interface according to any embodiment described herein.

[0207] Node 2030 may include a memory subsystem 2040. Memory 2040 may include a memory computation resource (comp) 2042, which represents the ability of memory 2040 to perform one or more memory computations. System 2000 supports remote memory operations, such as, but not limited to, those described elsewhere herein. Therefore, node 2030 may request a remote node to perform a memory computation, wherein the data used for the computation remains local to the executing node and is not sent via structure 2070 or from memory to the structure interface. In response to the execution of the memory computation, the executing node may provide the result to the requesting node.

[0208] Processor 2032 may include one or more individual processors. Each individual processor may include a single processing unit, a multi-core processing unit, or a combination thereof. A processing unit may include a main processor, such as, but not limited to, a CPU (Central Processing Unit), a peripheral processor (such as, but not limited to, a GPU (Graphics Processing Unit)), or a combination thereof. Memory 2040 may be or include memory devices and memory controllers.

[0209] The term "memory device" can refer to different types of memory. Memory devices generally refer to volatile memory technology. Volatile memory is memory whose state (and the data stored within it) is uncertain if power is interrupted. Non-volatile memory is memory whose state is deterministic even if power is interrupted. Dynamically volatile memory can refresh the data stored in the device to maintain its state. An example of dynamically volatile memory includes DRAM (Dynamic Random Access Memory) or variations thereof, such as, but not limited to, Synchronous DRAM (SDRAM). The memory subsystem described in this article is compatible with a variety of memory technologies, such as, but not limited to, DDR3 (Double Data Rate version 3, originally released by JEDEC (Joint Electron Device Engineering Committee) on June 27, 2007, currently version 21), DDR4 (DDR version 4, initial specification released by JEDEC in September 2012), DDR4E (DDR version 4, extended version, currently under discussion by JEDEC), LPDDR3 (Low Power DDR version 3, JESD209-3B, released by JEDEC in August 2013), and LPDDR4 (Low Power Double Data Rate (LPDDR) version 4). JESD209-4 (originally released by JEDEC in August 2014), WIO2 (Wide I / O2, JESD229-2, originally released by JEDEC in August 2014), HBM (High Bandwidth DRAM, JESD235, originally released by JEDEC in October 2013), DDR5 (DDR version 5, currently under discussion by JEDEC), LPDDR5 (currently under discussion by JEDEC), HBM2 (HBM version 2, currently under discussion by JEDEC) or combinations of other memory technologies, as well as technologies derived from or extended based on such specifications.

[0210] In addition to or as an alternative to volatile memory, in one embodiment, a reference to a memory device may refer to a non-volatile memory device whose state is deterministic even when power is interrupted. In one embodiment, a non-volatile memory device is a block-addressable memory device, such as, but not limited to, NAND or NOR technology. Therefore, the memory device may also include future-generation non-volatile devices, such as, but not limited to, three-dimensional cross-point (3DXP) memory devices, other byte-addressable non-volatile memory devices, or memory devices using chalcogenide phase change materials (e.g., chalcogenide glasses). In one embodiment, the memory device may be or include multi-threshold NAND flash memory, NOR flash memory, single-level or multi-level phase change memory (PCM) or switched phase change memory (PCMS), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), magnetoresistive random access memory (MRAM) incorporating memristor technology, or spin-transfer torque (STT)-MRAM, or any combination of the foregoing, or other memories.

[0211] In at least one embodiment, system 2000 may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0212] Figure 21An accelerated processing unit 2100 according to at least one embodiment is illustrated. The accelerated processing unit 2100 may include a processor based on the CDNA architecture of AMD Inc., Santa Clara, California, or other processors sharing at least some of the components described herein. The accelerated processing unit 2100 may include one or more accelerator complex dies (XCDs) 2104 for performing the operations described elsewhere in this document, such as, but not limited to, graphics processing and / or parallel processing and instruction-level parallel computing, including support for multiple precisions (INT8, FP8, BF16, FP16, TF32, FP32, and FP64) and sparse matrix data (i.e., sparsity). In some cases, the XCD may be referred to as a graphics computing die (GCD). The accelerated processing unit 2100 may include one or more complex computing dies (CCDs) 2106 for performing the operations described elsewhere in this document, such as, but not limited to, operations performed by a host processor. In some cases, the CCD may be referred to as a core complex or CCX, such as, but not limited to, the CCX used in AMD Ryzen processors. XCDs and CCDs can share any type of cache or memory (e.g., one or more memory cells 2102), or a cache or memory can be allocated to each XCD or CCD or group of XCDs or CCDs. For example, AMD Infinity Fabric within the package connects XCDs and CCDs to a shared AMD Infinity Cache 2108, and in some embodiments, to high-bandwidth memory (e.g., HMB3). Accelerated processing unit 2100 may include an AMD MI300a processor comprising three CPU die (or CCD) and six accelerator die (XCD) located on top of four input-output dies (IODs) that may be layered on a single silicon die (e.g., via AMD Infinity Fabric) and linked together to eight high-bandwidth DRAM stacks in a ring to form a superchip. For systems using only accelerators, the AMD MI300x processor replaces the CCD with two or more XCDs.

[0213] Accelerated processing unit 2100 may include one or more input / output (I / O) interfaces. For example, XCD 2104 and CCD 2106 may reside together on one or more input-output dies (IODs) 2110, which may include one or more I / O interfaces. IOD 2110 may include any number and type of I / O interfaces (e.g., PCI, PCI expansion (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices may be coupled to I / O interface 2170. The I / O interface of IOD 2110 may also be used to connect one or more accelerated processing units 2100, for example, in a server architecture.

[0214] Accelerated processing unit 2100 may include one or more memory units 2102 for storing instructions and other information for performing the operations described in other sections of this document. Memory units 2102 may include any volatile memory, such as, but not limited to, the memory types described in other sections of this document, and may include, for example, high-bandwidth memory (e.g., HMB3) or high-bandwidth DRAM. The memory associated with accelerated processing unit 2100 (e.g., memory unit 2102) may include system memory, which can be used for, for example, commands, instructions, and constants, as well as input and output. Memory unit 2102 may also include device memory, which can be used for storage and, for example, for commands, instructions, and constants, as well as input and output, as a return buffer, and for private data. Memory unit 2102 may be linked to one or more IODs 2110. L1 cache 2120 initiates a memory hierarchy including a shared L2 cache 2128 (e.g., within an XCD). AMD Infinity Cache™ is the last-level cache (LLC) located on the active I / O die (IOD). CCD 2106 and XCD 2104 can have independent or shared memory. AMD Infinity architecture and AMD Infinity Fabric™ technology enable consistent, high-throughput unification of GPU and CPU chip technologies (such as XCD, CCD, and / or CCX) with memory (such as stacked HBM3 memory) in a single device and across multiple device platforms.

[0215] like Figure 21As shown, XCD 2104 may include a set of shared global resources 2130, which may include a hardware scheduler 2132 and an asynchronous compute engine (ACE) 2124. The ACE 2124 sends instructions (e.g., compute shader workgroups) to compute units (CUs or cores) 2134. ACEs 2124 (e.g., four) may each be associated with CUs 2134 (e.g., 40 CUs), and some CUs 2134 may be disabled for yield management. CUs 2134 may have dedicated caches or shared caches (e.g., L2 caches) 2128 for consolidating all memory traffic on a single die. CU2134 may include threads and parallel processor cores, including instruction fetching and scheduling using a scheduler (S) 2112, a matrix core unit (MCU) 2116, and a shader core (SC) 2118 (e.g., execution units for scalar, vector, and matrix data types), and a load / store pipeline with an L1 cache 2120 and a local data share (LDS) 2114. The local data share may include, for example, a sticky-note RAM with built-in arithmetic capabilities, allowing data sharing between threads in a workgroup. Instruction cache 2140 (e.g., for storing and providing instructions for performing the operations described elsewhere in this document) and constant cache 2138 may be connected to one or more CUs and may be shared between two CUs. Matrix core 2116 can handle various data types, such as, but not limited to, INT8, FP8, FP16, BF16, and TF32 data types. Accelerated processing unit 2100 may include computation units 2134, which may be arranged in an array format, such as as a data parallel processor (DPP) array. The hyper-threaded dispatch processor 2142 can communicate with the compute unit 2134, and the command processor 2144 can read commands written by the host to memory-mapped registers in the system memory address space (not shown). When a command is completed, the command processor 2144 can send a hardware-generated interrupt to the host processor (e.g., a CCD). The memory controller 2136 can also directly access all device memories and system memory regions specified by the host. To satisfy read and write requests, the memory controller 2136 can perform the functions of a direct memory access (DMA) controller, including calculating the memory address offset based on the format of the requested data in memory. For example, one or more APIs described herein can be compiled into instructions that can be stored in the instruction cache 2140, then fetched by the instruction fetch logic in the processor 2140, decoded by the processor decoder or equivalent, scheduled (e.g., sequentially or out of order) by the scheduler or equivalent for execution, executed by the execution logic or equivalent, reordered, and then retired by the retirement logic or equivalent.The API (and / or compiled instructions including the API) may be stored in any storage device, either inside or outside the processor 2100 (e.g., in a cache and / or memory). The results of the API may be stored in storage devices, either inside or outside the processor 2100, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0216] Applications may include programs running on the main processor (e.g., a CCD) and programs running on one or more XCDs (referred to as kernels). Programs can be controlled by host commands that set internal base addresses and other configuration registers, specify data fields on which the accelerator processing unit 2100 can run, invalidate and flush caches on the accelerator processing unit 2100, and cause the accelerator processing unit 2100 to begin executing a program. A kernel can be referred to as a program executed by the accelerator processing unit 2100. Kernels can execute independently on each work item or as a group of work items, referred to as a wavefront, which can execute kernels on all (e.g., 64) work items in a single pass. The computation unit 2134 may include: a scalar arithmetic logic unit (ALU) that can operate on a single value for each wavefront (shared by all work items); a vector ALU that can operate on a unique value for each work item; a local data share 2114 that allows work items within a workgroup to communicate and share data; a scalar memory (not shown) that can transfer data between the scalar general-purpose registers (SGPRs) and memory via cache; and a vector memory that can transfer data between the vector general-purpose registers (VGPRs) and memory, including sampling of texture maps. Kernel control flow can be manipulated using scalar ALU instructions, which may include if / else statements, branches, and loops. Scalar ALU (SALU) and memory instructions can operate on the entire wavefront and operate on one or more SGPRs. Vector memory and ALU instructions can operate on all work items in the wavefront simultaneously.

[0217] In at least one embodiment, the acceleration processing unit 2100 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. The one or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0218] Figure 22Processor 2200 is shown, such as, but not limited to, a Zen architecture-based processor (e.g., Zen1, 2, 3, 4, 5, or other architectures) from AMD Inc. of Santa Clara, California, or other processors that share at least some of the components described herein. Processor 2200 includes one or more CPU dies 2202(1)-2202(N), where N is any integer greater than 1. CPU die 2202 may include any number of processor cores 2216 (e.g., for performing any operations described elsewhere herein) and any number of cache memories (e.g., for storing instructions and other information to perform any operations described elsewhere herein). For example, L2 cache unit 2218 may be coupled to processor core 2216, and processor core 2216 may share and / or be individually coupled to L2 cache unit 2218. Processor core 2216 may be coupled to L3 cache 2222 and / or a shared L3 cache, which may be the lowest level cache (LLC) 2222 used to access data and other information used by processor core 2216. One or more processor cores 2216 and one or more L2 cache units 2218 may be included in a core complex (CCX) 2220, which may include (e.g., 32MB) a shared cache (e.g., L3 cache 2222). Core complex 2220 may be manufactured onto a die (CCD or CPU die) 2202. For example, up to 12 core complexes 2220 may be configured in a processor along with 8 CPU dies 2202, thereby providing up to 96 processor cores 2216 for processor 2200. For example, a "Zen 4c" core complex 2220 may include up to 8 cores 2216 and a shared 16MB L3 cache 2222. Two core complexes of these core complexes 2220 can be combined onto a single CPU die 2202, resulting in 16 cores per die and a total of 32MB of L3 cache per die 2222. Up to eight CPU dies 2202 can be combined with I / O units 2204 to provide up to 128 processor cores 2216 for the CPU. Up to four "Zen 4c" dies mentioned above can be combined to provide up to 64 processor cores 2216 for the CPU.

[0219] Processor 2200 may include various configurations for input / output operations, which will be further described herein. I / O unit 2204 may include one or more memory controllers 2206 capable of managing the memory usage of processor 2200 (e.g., DDR5 memory). I / O unit 2204 may include one or more SATA disk controllers for managing storage device 2212, and one or more Compute Express Link (CXL™) 1.1+ memory controllers 2214 for providing CPU-to-device and CPU-to-memory connectivity and can be flexibly assigned to specific functions during server design. I / O unit 2204 may include PCIe controller 2208 for connecting peripherals and other components connected to processor 2200. I / O unit 2204 may also include USB port 2210 for connecting to other components separate from processor 2200. CPU die 2202 may support any number of connections to I / O unit 2204, for example, one or two connections. As shown in the figure, I / O unit 2204 may include the components further described herein, and I / O unit 2204 may be an I / O die accommodating several different components. Memory controller 2206, PCIe controller 2208, USB port 2210, SATA controller 2212, and / or CXL controller 2214 may be individually integrated into any location within processor 2200, or integrated in any group or combination.

[0220] Processor 2200 may include an Infinity Fabric 2224 interconnect (which may be similar to or based on a PCIe architecture) that provides connectivity between CPUs (e.g., CPU dies 2202(1)-2202(N)), graphics processor 2226, inference engine 2232, and other components in a multi-chip architecture (e.g., security processor 2228 and I / O unit 2204). One or more AMD Infinity Fabric™ interconnects 2210 may be connected to CPU dies 2202(1)-2202(N) and used as connections between CPUs. One or more Infinity Fabric interconnects 2210 may connect each CPU die 2202 to an I / O unit 2210.

[0221] In at least one embodiment, processor 2200 may include a central processing unit (CPU) and other related hardware and software described above and further herein. Processor 2200 may also include a graphics processor 2226. Graphics processor 2226 may be used for image generation and processing, as well as other computations and operations described further herein. Graphics processor 2226 may be based on AMD's RDNA 3 or 3.5 architecture, located in Santa Clara, California. Graphics processor 2226 may include a graphics computing die (GCD) and a memory cache die (MCD). GCD may include any number of computing units (CUs) for graphics or other processing, such as operations performed by an arithmetic logic unit (ALU) described further herein. Graphics processor 2226 may include an L2 cache available for use by the computing units. MCD (not shown) may include any number of memory cells and may include a cache (e.g., an L3 cache) and a memory interface for coupling to memory (e.g., memory 2242(1)-(N), where N is an integer). Components within the graphics processor 2226 can be connected using various methods, such as using Infinity Fabric 2224 interconnects, either internally or externally to the graphics processor 2226.

[0222] Inference engine 2232 can provide neural processing capabilities to processor 2200 for computational processes used in neural networks, deep learning, and other AI-related operations, which will be further described herein. Processor 2200 may include: a security processor 2228 for managing the security of processor 2200; a display controller 2230 for controlling the display; a system management unit 2234 for managing and operating some or all components on processor 2200; a multimedia engine 2236 for audio and video operations; a fusion controller hub 2238 for managing USB, SATA, and PCIe connections to the processor; and a sensor fusion hub 2240 for managing sensors, such as accelerometers. Processor 2200 may also include memory 2242(1)-(N), where N is any integer. Memory may include different memory types, such as LPDDR5 and / or DDR5, or other memory described elsewhere herein.

[0223] To perform the operations further described herein, processor 2200 may include an execution pipeline including a front-end that may include a cache for storing instructions (e.g., an L1 cache) (not shown). A branch predictor may modify the instruction stream. Instructions may be decoded by a decoder, dispatched to a back-end for execution, and renamed. For example, the instruction fetch and decode pipeline may be dispatched to integer or floating-point execution operations, which may be scheduled by a scheduler and passed to vectors and / or general-purpose registers. Floating-point multipliers and / or addition operations may be processed, and an arithmetic logic unit (ALU) may also be used to perform computations, such as arithmetic and logical operations. The output of the computation unit may be coupled to a load / store queue that may be connected to a cache, such as an L1 cache and / or an L2 cache.

[0224] With respect to processor 2200 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents (e.g., AVX-512 instructions based on a SIMD model). These instructions or equivalents may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) internal or external to processor 2200. The results of the APIs may then be stored in storage devices internal or external to processor 2200, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0225] In at least one embodiment, processor 2200 may include one or more circuits configured to execute an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0226] Figure 23An example of a processing core 2300 is shown, which may implement an Arm architecture (e.g., v9.0-A) or another processor sharing at least some of the components described herein. The Neoverse™ V2 core 2300 may be implemented within a DynamIQ shared unit (DSU) cluster via a DSU-110 interconnect 2354 for connecting one or more cores, for example, for parallel processing. The Neoverse™ V2 core may be implemented as a single core within a DSU cluster configured for direct interconnection, with or without L3 cache, listener filter, or listener control unit (SCU) logic (not shown). The Neoverse™ V2 core may include a CPU bridge 2352 for connecting the core 2300 to the DSU-110 interconnect, which may also connect the core 2300 to an external memory system and the remainder of the on-chip system. The L1 instruction memory system 2302 can fetch instructions from the instruction cache 2304 and deliver the instructions (e.g., one or more compileable APIs described herein) to the instruction decoding unit 2310, for example, to perform some or all of the operations described above or elsewhere herein. The L1 instruction memory system 2302 may include the L1 instruction cache 2304 (e.g., with 64-byte cache lines), the L1 instruction translation back buffer (TLB) 2306 (e.g., natively supporting 4KB, 16KB, 64KB, and 2MB page sizes), and the macro operation cache (MOP) 2308 (e.g., a 1536-entry, 4-way skewed associated L0 MOP cache), which may include decoded and optimized instructions for higher performance. The instruction decoding unit 2310 can decode AArch64 instructions into its internal format. The register renaming unit 2312 can perform register renaming to facilitate out-of-order execution and dispatch the decoded instructions to various issue queues. Instruction issuing unit 2314 can control when decoded instructions are dispatched to the execution pipeline, and it can include an issuing queue for storing instructions to be dispatched to the execution pipeline. Integer execution pipeline 2316 can be included in the execution pipeline and includes integer execution unit 2318 capable of performing arithmetic and logical data processing operations. Vector execution unit 2320 can be included in the execution pipeline and can execute advanced SIMD and floating-point arithmetic (FPU) 2322, execute Scalable Vector Extension (SVE) and Scalable Vector Extension 2 (SVE2) instructions 2324, and can also selectively execute cryptographic instructions 2326. Advanced SIMD may include a media and signal processing architecture that primarily adds instructions for audio, video, 3D graphics, image, and speech processing. The floating-point architecture provides support for single-precision and double-precision floating-point operations. L1 data memory system 2330 can execute load and store instructions, as well as servicing memory coherence requests.The L1 data memory system 2330 may include an L1 data cache 2332 and a fully associative L1 data TLB 2334, natively supporting 4KB, 16KB, and 64KB page sizes and 2MB and 512MB block sizes. The memory management unit (MMU) 2328 provides fine-grained memory system control through a set of virtual-to-physical address mappings and memory attributes, which can be stored in a translation table and saved to the TLB 2334 after address translation. The L2 memory system 2336 may include an L2 cache 2338 and can be connected to the DSU-110 2354 via an asynchronous CPU bridge 2352. The Neoverse™ V2 core 2300 supports a range of debug, test, and tracing options, including a tracing unit 2342, a tracing buffer 2340, and an embedded logic analyzer (ELA) 2348. The Neoverse™ V2 core 2300 implements a Statistical Analysis Extension (SPE) 2344 to provide a statistical view of the performance characteristics of the executed instructions. Software writers can leverage these views to optimize code for better performance. A Performance Monitoring Unit (PMU) 2346 provides a performance monitor that can be configured to collect statistics on the operation of each core and memory system. This information can be used for debugging and code analysis. The Generic Interrupt Controller (GIC) CPU interface 2350, when integrated with external allocator components, serves as a resource for supporting and managing interrupts in a clustered system. Within the cluster, there can be a CPU bridge 2352 between each Neoverse™ V2 core 2300 and the DSU-110 2354. The CPU bridge 2352 controls buffering and synchronization between the core 2300 and the DSU-110 2354. The CPU bridge 2352 can be asynchronous to allow each core 2300 to use different frequency, power, and area implementation points. The CPU bridge 2352 can operate synchronously without affecting other interfaces, such as, but not limited to, asynchronous debug and trace interfaces.

[0227] In at least one embodiment, core 2300 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0228] Figure 24 One or more chips including one or more tensor processing units (TPUs) 2400 are shown according to at least one embodiment. Figure 24The TPU 2400 may include an application-specific integrated circuit (ASIC), for example, for performing some or all of the operations described above or elsewhere herein, such as, but not limited to, machine learning workloads that accelerate the execution of matrix operations. The TPU 2400 may be an ASIC from Alphabet Corporation, Mountain View, California. The cloud TPU includes a cloud service that enables the TPU to be used as a scalable resource for processing instructions, such as, but not limited to, machine learning workloads that can run on frameworks such as, but not limited to, TensorFlow, PyTorch, and JAX.

[0229] Chip 2400 may include any number of TPUs, which may include tensor cores 2406. Tensor core 2406 may include one or more core sequencers 2408, vector processing units (VPUs) 2410, matrix multiplication units (MXUs) 2412(A)-2414(N) (where N is any integer greater than 1), and transpose permutation units 2416. Core sequencer 2408 may fetch instructions (e.g., VLIW (Very Long Instruction Word)) from the instruction memory (Imem) of core 2406, perform scalar operations using scalar data memory (Smem) and scalar registers (Sregs) (not shown), and forward vector instructions to vector processing units (VPUs) 2410. For example, an instruction may initiate eight operations: two scalar operations, two vector ALU operations, vector loading and storing, and queuing data into matrix multiplication and transpose units and a pair of slots for queuing data from them. The VPU2410 can perform vector operations using a large on-chip vector memory (Vmem) and vector registers (Vregs). The VPU 2410 can stream data to or from the MXU via a decoupled FIFO. The VPU 2410 can collect and distribute data to the Vmem using both data-level parallelism (2D matrix and vector function units) and instruction-level parallelism (8 operations per instruction). Large two-dimensional matrix multiplication units (MXUs) 2412(A)–2412(N) can, for example, use a systolic array to reduce area and power consumption, and use a large, software-controlled on-chip memory instead of a cache. The transpose-reduction-permute unit 2416 can perform (e.g., 128x128) matrix transpose, reduction, and permute on the VPU 2410 channels. A high-bandwidth memory 2404 can be used for on-chip applications and can be coupled to a host queue 2402, for example, via PCIe. One or more chips 2400 can be connected together for computing. For example, one or more chips 2400 can be connected to form a torus, such as a 2D torus. Chips 2400 can also include any number (e.g., 4) of inter-core interconnect (ICI) links 2418, which can enable direct connections between chips to form a supercomputer.

[0230] For any processor in chip 2400 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device (e.g., cache and / or memory) external to or internal to any processor in chip 2400. The results of the APIs may then be stored in any storage device internal to or external to any processor in chip 2400, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0231] In at least one embodiment, chip 2400 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0232] Figure 25A vector processor according to at least one embodiment is illustrated. The vector processor 2500 may support the RISC-V standard. The vector processor 2500 may include one or more cores 2510 (e.g., scalar units) and one or more vector processing units (VPUs) 2542 (e.g., vector units), which may, for example, perform some or all of the operations described above or elsewhere herein. The core 2510 may include an Andes Custom Extension (ACE) 2516, which may be used, for example, to deliver custom instructions to the processor 2500 via an ACP 2538. The core 2510 may include a 1-cycle multiplier and a 1-cycle instruction / data local memory (ILM / DLM) for improving parallelism by allowing simultaneous instruction fetching and data access. A memory management unit (MMU) 2524 manages system memory and cache, and provides branch execution, instruction pair issuance, L1 instruction / data cache, and local memory storage. The core 2510 may include a physical memory protection and programmable physical memory attribute unit (PMP / PPMA) 2522. Core 2510 may include a digital signal processor (DSP) 2528 and a floating-point unit (FPU) 2526, as well as a load-memory unit (LSU) 2532 for interaction with memory hierarchies (D$ 2534 and I$ 2530). Core 2510 may include a branch prediction unit 2518 and a multiplier unit 2520.

[0233] The vector processing unit (VPU) 2542 may include one or more vector function units (FU) 2546(A)-2546(N) (which may be linked together for parallel processing), a separate memory path for loading / storing RISC-V vectors (RVVs) via the ACE-RVV 2548 and the AndesStreaming port (ASP) 2544, and a vector load / store unit (VLSU) 2550.

[0234] The vector processor 2500 may include bus interfaces such as, but not limited to, a cacheable L2 cache port 2556, a non-cacheable MMIO port 2554, an input-output coherence port (IOCP) 2558 for a cacheless bus master, a local memory access port for accessing the ILM / DLM 2512 (which can be coupled to the SRAM 2506) and the high-bandwidth vector memory (HVM) 2536, and a shared peripheral port (SPP) 2552 for external peripherals. Other memory ports include the LM slave port AXI 2502, the HVM subordinate port AXI 2504, the MEM (AXI) 2562, and the AXI 2560. The trace I / F 2514 can be captured, encoded, and transmitted off-chip via the Inst. trace I / F 2508 (e.g., a record of executed processor instructions). Software tools can use the Inst. trace I / F 2508 to reconstruct the exact execution sequence of a program.

[0235] For any processor in processor 2500 and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. APIs (and / or compiled instructions including APIs) may be stored in any storage device external to or internal to processor 2500 (e.g., in cache and / or memory). The results of the APIs may then be stored in storage devices internal to or external to processor 2500, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0236] In at least one embodiment, the vector processor 2500 may include one or more circuits for executing an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. The one or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0237] Figure 26A A schematic diagram of an example multi-core tiled processor microarchitecture is shown. Figure 26A Multi-core tiled processors in a system can include language processing processors. For example... Figure 26AAs shown, each “tile” in the processor architecture is a processing element bundled together using an on-chip network (NoC), which can be used to perform some or all of the operations described above or elsewhere in this document. For example, each tile may have an instruction dispatch 2604 and integer (INT) units 2606 and floating-point (FP) units 2608, a load-memory unit (LSU) 2612 for engaging with a memory hierarchy (data cache (D$) 2610 and instruction cache (I$) 2614), and a network (NET) interface 2616 for communicating with other tiles. Some tiles in processor 2600 may include a memory controller 2602 for managing and controlling memory, as further described herein. Processor 2600 may have a functionally sliced ​​architecture. Processor 2600 may reside on an application-specific integrated circuit (ASIC). Figure 26A The layout of an ASIC can be represented. Processor 2600 may include a coprocessor designed to execute instructions for a predictive model. A predictive model refers to any model configured to make predictions based on input data. The predictive model can use a classifier for classification predictions. The predictive model can be a machine learning model, such as, but not limited to, a tensor flow model, and processor 2600 is a tensor flow processor.

[0238] The processor 2600 can use different microarchitectures, and these microarchitectures will Figure 26B Each tile in the process represents a separate functional unit. Conversely, the functional tiles 2624 of the processor 2600 can be aggregated into multiple functional processing units (hereinafter referred to as "slices") 2604, each corresponding to a specific functional type (e.g., FP / INT 2618, NET 2620, MEM 2622). For example, as... Figure 26B As shown, each slice may correspond to a row of functional tiles extending in a north-south direction. Furthermore, the processor 2600 may also include communication channels for carrying data between tiles in different slices, each communication channel extending horizontally in an east-west direction. Each communication channel may be connected to each slice 2604 of the processor 2600.

[0239] The slices 2604 of processor 2600 may each correspond to different functions and may include arithmetic logic slices (e.g., FP / INT 2618), channel switching slices (e.g., NET 2620), and memory slices (e.g., MEM 2622). Arithmetic logic units can perform one or more arithmetic and / or logical operations on data received via communication channels to generate output data. Examples of arithmetic logic units may be matrix multiplication units and vector multiplication units. Memory slices include memory units that store data. Memory slices can provide data to other slices via communication channels. Memory slices can also receive data from other slices via communication channels. Channel switching slices can configurably route data from one communication channel to any other communication channel. For example, data from a first channel can be provided to a second channel via a channel switching slice. In some embodiments, a channel switching slice can be implemented as a crossbar switch. Each slice 2604 also includes its own instruction queue (not shown) for storing instructions and an instruction control unit (ICU) for controlling instruction execution. Instructions in a given instruction queue can only be executed by a tile in its associated functional slice and not by other slices of processor 2600.

[0240] By arranging the tiles of processor 2600 into different functional slices 2604, the on-chip instruction and control flow of processor 2600 can be separated from the data flow. For example, according to some embodiments, Figure 26B One of the arrows illustrates the instruction flow within the processor architecture. According to at least one embodiment, Figure 26B Another arrow in the diagram illustrates the data flow within the processor architecture. As shown, instruction and control flow can move across the tiles of processor 2600 in a first direction (e.g., north-south along the length of a functional slice, as indicated by the first arrow), while data flow can move across the tiles of processor 2600 in a second direction (e.g., east-west across a functional slice, as indicated by the second arrow), which is perpendicular to the first direction.

[0241] Different functional slices of processor 2600 can correspond to MEM 2622 (memory), VXM (vector execution module), MXM (matrix execution module), NIM (numerical interpretation module), and SXM (swapping and permutation module). Each slice can include N tiles, all of which can be controlled by the same instruction control unit (ICU) (not shown). Each slice can operate completely independently and can only be coordinated using barrier-like synchronization primitives or by the compiler using tractable determinism. Each tile of processor 2600 can correspond to an execution unit organized as ×M SIMD tiles. For example, each tile of the on-chip memory of processor 2600 can be organized to atomically store L-element vectors. Therefore, MEM slices with N tiles can work together to store or process large vectors (e.g., with a total of N×M elements).

[0242] Tiles within a slice can execute instructions in an "interleaved" manner, where instructions can be issued tile-by-tile within the slice over N cycle periods. Functional slices can be physically arranged on the chip to allow for efficient data flow for pipelined execution over hundreds of cycles for common patterns. The data flow can perform a single "u-turn" (direction change) corresponding to a single matrix operation before being written back to memory; in some embodiments, a particular data flow can change direction multiple times before writing the resulting data back to memory (due to multiple matrix and vector operations).

[0243] When using a processor 2600 with a function slice architecture (e.g., TSP), the TSP compiler (not shown) generates an explicit plan of how the processor 2600 can execute programs (e.g., microprograms). The compiler can specify when each operation will be executed, which function slices will perform the work, and which STREAM registers will hold operands. The compiler can maintain a high-fidelity (cycle-accurate) model of the processor 2600 (e.g., TSP) hardware state so that microprograms can coordinate data flow.

[0244] Processor 2600 (e.g., TSP) can use a web-hosted compiler that takes a model (e.g., an ML model, such as but not limited to a TensorFlow model) as input and issues a proprietary stream of instructions for processor 2600 (e.g., TSP). The compiler is responsible for coordinating the control and data flow of the program and specifying any instruction-level parallelism by explicitly bundling instructions that can and should be executed concurrently so that they can be dispatched together. The main hardware architecture includes the architecture-visible streaming register file (STREAM), which will be described in more detail below, and acts as a conduit for operands to flow from MEM slices (e.g., SRAM) to functional slices (and vice versa).

[0245] The MEM 2622 of the processor 2600 can be used as: (1) a storage for model parameters, microprograms, and data on which they operate; and (2) an on-chip network (NoC) for transferring data operands from the MEM to functional slices and returning computation results to the MEM. In some embodiments, the on-chip memory may consume approximately 75% of the chip area of ​​the processor 2600. In some embodiments, the on-chip memory of the MEM tile may include SRAM instead of DRAM due to the bandwidth requirements of the processor 2600. The on-chip memory capacity of the processor 2600 may be determined by: (i) the number of ML models that can reside on the chip simultaneously, (ii) the size of any given model, and (iii) partitioning of large models for adaptation to a multi-chip system. In some embodiments, the MEM system of the processor 2600 may provide multiple memory slices organized into two distinct hemispheres (referred to as “MEM WEST” and “MEM EAST”, respectively).

[0246] The memory slices in each hemisphere can be mirrored such that the slices are physically numbered {0,...L} in the eastern hemisphere and {L,...0} in the western hemisphere, such that memory slice 0 in each hemisphere corresponds to the slice of the VXM slice closest to the hemisphere, where each hemisphere comprises L slices. Data transfer towards the chip center can be referred to as inward, while data transfer towards the outer edge of the chip (easternmost or westernmost) can be referred to as outward. Although the memory hemispheres of processor 2600 can be referred to as east and west, it is understood that other names may be used to refer to different memory hemispheres in other embodiments.

[0247] In some embodiments, streaming register files (referred to as STREAMS) transfer operands and results between the SRAM of the MEM slices of the processor 2600 and the functional slices. In some embodiments, multiple MEM slices (e.g., 2 to 10 adjacent MEM slices) can be physically organized into sets. Each slice set can be located between a pair of STREAMS register files, allowing each slice to read from or write to the STREAMS registers in either direction. By placing the STREAMS register files between sets of MEM slices, the number of cycles required to transfer data operands across hemispheres can be reduced (e.g., reduced by a factor corresponding to the number of slices per set). The number of slices per set can be configured based on the distance of data transfer within a single clock cycle.

[0248] for Figure 26A Any processor and any components described above or elsewhere herein, one or more APIs or equivalents described herein may, for example, be compiled into instructions or equivalents that may be fetched by instruction fetching logic or equivalents, decoded by processor decoder or equivalents, scheduled (e.g., sequentially or out of order) for execution by scheduler or equivalents, executed by execution logic or equivalents, reordered, and then retired by retirement logic or equivalents. The API (and / or compiled instructions including the API) may be stored in any storage device internal or external to processor 2600 (e.g., in cache and / or memory). The results of the API may then be stored in storage devices internal or external to processor 2600, including registers, DRAM, flash memory, SRAM, cache, or other memory equivalents.

[0249] In at least one embodiment, processor 2600 may include one or more circuits configured to execute an application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be software-configured to execute the application programming interface (API) to pause and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0250] Software Structure The following figures illustrate, without limitation, examples of software structures for implementing at least one embodiment.

[0251] Figure 27A software stack for a programming platform according to at least one embodiment is illustrated. The programming platform may include a platform for accelerating computation instructions by utilizing hardware on a computing system. In at least one embodiment, software developers can access the programming platform through libraries, compiler instructions, and / or extensions to the programming language. The programming platform may be CUDA, Radeon Open Computing Platform (“ROCm”), OpenCL (OpenCL™ developed by the Khronos Group), SYCL, or Intel oneAPI.

[0252] The software stack 2700 of the programming platform can provide an execution environment for application 2701. Application 2701 may include any computer software capable of being launched on the software stack 2700. Application 2701 may include artificial intelligence (“AI”) / machine learning (“ML”) applications, high-performance computing (“HPC”) applications, virtual desktop infrastructure (“VDI”), or data center workloads.

[0253] Application 2701 and software stack 2700 run on hardware 2708. Hardware 2708 may include one or more GPUs, CPUs, FPGAs, AI engines, and / or other types of computing devices supporting programming platforms. Software stack 2700 may be vendor-specific and compatible only with vendor-specific devices, such as CUDA, ROCm, OneAPI, OpenCL, or other implementations. Hardware 2708 may include a host connected to one or more devices that can be accessed via application programming interface (“API”) calls to execute computational instructions. In at least one embodiment, the devices within hardware 2708 may include GPUs, FPGAs, AI engines, or other computing devices (but may also include CPUs) and their memory, while the host within hardware 2708 may include CPUs (but may also include computing devices) and their memory. For any hardware 2708 described above or elsewhere herein, the one or more APIs described herein may, for example, be compiled into instructions that may be fetched by instruction fetching logic, decoded by a processor decoder, scheduled (e.g., sequentially or out of order) for execution by a scheduler, executed by execution logic, reordered, and then retired by retirement logic. The API (and / or compiled instructions including the API) may be stored in any storage device (e.g., cache and / or memory) internal or external to the hardware 2708. The results of the API may be stored in storage devices internal or external to the hardware 2708, including registers, DRAM, flash memory, SRAM, cache, or other memory. One or more APIs described herein may receive calls. One or more APIs described herein may communicate with a library or a portion of a library to perform the function described by the call. One or more APIs described herein may receive calls and communicate with a library or a portion of a library to perform the function described by the call.

[0254] The software stack 2700 of the programming platform may include multiple libraries 2703, a runtime 2705, optional drivers / interfaces 2707, and device kernel drivers 2708. Each library 2703 may include data and programming code that can be used by computer programs and utilized during software development. Library 2703 may include pre-written code and subroutines, classes, values, type specifications, configuration data, documentation, help data, and / or message templates. Library 2703 may include functionality optimized for execution on one or more types of devices. Library 2703 may include functionality for performing mathematical, deep learning, and / or other types of operations on the device. Library 2703 may be associated with corresponding APIs 2702, which may include one or more APIs for exposing the functionality implemented in library 2703. A processor (e.g., CPU, GPU) may execute, call, or otherwise use one or more APIs to determine kernel priority. For example, a first kernel (e.g., a parent kernel) may launch a second kernel (e.g., a child kernel), and the processor may use the second kernel to launch an additional kernel (e.g., a grandchild kernel) independent of the first kernel. The processor can execute APIs or call APIs from memory to support dynamic stream priorities (e.g., updating priorities when performing operations using streams). For example, when the processor executes the API, it allows the programmer to copy stream priorities from one stream to one or more other streams.

[0255] Software stack 2700 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to set the stream's priority at any time after the stream is created. Software stack 2700 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to obtain the stream's current priority, where the priority is one of several attributes of the stream. Software stack 2700 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to obtain the stream's current priority as a single attribute. Software stack 2700 may include APIs that support dynamic stream prioritization (e.g., updating priority while performing operations on the stream), allowing programmers to start the kernel to perform operations on the stream at a set priority, which may be different from the stream priority. Software stack 2700 may include an API for indicating whether an object (e.g., a thread synchronization object, such as, but not limited to, a barrier) tracks whether all data movement operations of a set of threads running on the GPU have a specified state after a specified time period, wherein the specified state may be a state indicating that data has been moved and is ready for use, and is specified using expected parity values ​​as input to the API.

[0256] Software stack 2700 may include one or more APIs for updating the kernel. The processor may execute APIs or call APIs from memory to update existing APIs, thereby supporting a context-independent kernel. This allows programmers to add kernel nodes to a graph without a graph context, so that the graph context can be dynamically associated with the kernel at runtime. Software stack 2700 may include one or more APIs that allow programmers to obtain kernel identifiers and graph contexts as separate parameters from kernel nodes, thereby enabling parameter retrieval from both the kernel and the context-independent kernel. Software stack 2700 may include one or more APIs that use parallel processors (e.g., but not limited to one or more graphics processing units) to initiate instruction graphs (e.g., instruction maps) and execute one or more instruction maps (e.g., including one or more programs).

[0257] Software stack 2700 may include one or more APIs for associating one or more instructions with one or more memory sorting operations (such as, but not limited to, fence or memory barrier operations). Instructions may be associated with one or more domains, causing memory sorting operations to execute in relation to one or more specific domains without interfering with instructions in other domains. APIs may indicate that a thread has reached (e.g., reached a thread synchronization barrier) or completed a certain phase of work associated with an asynchronous data movement operation on the GPU. Software stack 2700 may include one or more APIs that allow programmers to manually indicate an expected transaction count when a thread completes a certain phase of work; this transaction count can be used to update an object used to track whether all data movement operations for a set of threads have been completed.

[0258] Application 2701 can be written as source code and then compiled into executable code, as described below. Figure 28 and Figure 29 As discussed in more detail, the executable code of application 2701 can run, at least partially, within the execution environment provided by software stack 2700. During the execution of application 2701, code that needs to run on the device (rather than the host) may be encountered. In this case, runtime 2705 can be invoked to load and launch the required code on the device. Runtime 2705 can include any technically feasible runtime system capable of supporting the execution of application 2701.

[0259] Runtime 2705 can be implemented as one or more runtime libraries associated with the corresponding API (which is shown as API 2704). One or more such runtime libraries may include functions for memory management, execution control, device management, error handling, and / or synchronization, etc. Memory management functions may include functions for allocating, dealing with, and copying device memory, as well as functions for transferring data between host memory and device memory. Execution control functions may include functions for starting functions on the device (sometimes referred to as "kernels" when the function is a global function that can be called from the host) and setting attribute values ​​in buffers maintained by the runtime libraries so that a given function can be executed on the device.

[0260] Runtime libraries and the corresponding API 2704 can be implemented in any technically feasible manner. One (or any number) APIs can expose a set of low-level functions for fine-grained control of the device, while another (or any number) APIs can expose a set of high-level functions for such functions. High-level runtime APIs can be built on top of low-level APIs. One or more runtime APIs can be language-specific APIs, which can be layered on top of language-independent runtime APIs.

[0261] Optional drivers or interfaces 2707 can be implemented, for example, for CUDA and ROCm implementations, which will be described further below. Optional drivers / interfaces 2707 can be associated with optional driver or interface APIs, such as, but not limited to, the CUDA and / or ROCm APIs.

[0262] One or more processors disclosed in the “processing system” may execute, access, or otherwise use the software stack 2700. For example, the system-on-a-chip 1400, parallel processor 1500, graphics multiprocessor 1534, processor 1600, processor 1700, accelerator 1800, neuromorphic processor 1905, supercomputer 2000, acceleration processing unit 2100, processor 2200, processor 2300, tensor processing unit 2400, processor 2500, and language processing unit 2600 may execute, use, call, or otherwise implement (e.g., by accessing memory) one or more APIs included in the software stack 2700.

[0263] Device kernel driver 2708 can be configured to facilitate communication with underlying devices. Device kernel driver 2708 provides low-level functionality for APIs (such as, but not limited to, API 2704) and / or other software. Device kernel driver 2708 can be configured to compile intermediate representation (“IR”) code into binary code at runtime. For CUDA or other implementations (such as, but not limited to, ROCm, OneAPI, or OpenCL), device kernel driver 2708 can compile non-hardware-specific parallel thread execution (“PTX”) IR code into binary code for a specific target device at runtime (and cache the compiled binary code), which is sometimes referred to as “finalized” code. Doing so allows the finalized code to run on a target device that may not have existed when the source code was initially compiled into PTX code. Alternatively, device source code can be compiled into binary code offline without device kernel driver 2708 compiling the IR code at runtime.

[0264] Processors described elsewhere in this document (e.g., but not limited to) Figure 14-26B The processor (in the document) may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2700) to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0265] According to at least one embodiment, Figure 27 The software stack 2700 can execute in a CUDA implementation. The CUDA software stack 2700, on which application 2701 can be launched, may include CUDA libraries 2703, CUDA runtime 2705, CUDA drivers 2707, and device kernel drivers 2708. The CUDA software stack 2700 can execute on hardware (e.g., a graphics multiprocessor 1534), which may include a CUDA-enabled GPU developed by NVIDIA Corporation, located in Santa Clara, California.

[0266] Application 2701, CUDA Runtime 2705, and Device Kernel Driver 2708 can perform the functions described above and elsewhere in this document. CUDA Driver 2707 may include a library (libcuda.so) that implements CUDA Driver API 2706. Similar to CUDA Runtime API 2704, implemented by the CUDA Runtime Library (cudart), CUDA Driver API 2706 exposes functions for memory management, execution control, device management, error handling, synchronization, and / or graphics interoperability. The difference between CUDA Driver API 2706 and CUDA Runtime API 2704 is that CUDA Runtime API 2704 simplifies device code management by providing implicit initialization, context (similar to processes), and module (similar to dynamically loaded libraries) management. Compared to the high-level CUDA Runtime API 2704, CUDA Driver API 2706 can serve as a low-level API, providing finer-grained device control, particularly in terms of context and module loading. CUDA driver API 2706 can expose context management functions not exposed in CUDA runtime API 2704. CUDA driver API 2706 can also be language-agnostic, supporting technologies such as OpenCL in addition to CUDA runtime API 2704. Furthermore, development libraries, including CUDA runtime 2705, can be considered separate from driver components, including user-mode CUDA driver 2707 and kernel-mode device driver 2708 (sometimes also called the "display" driver).

[0267] CUDA library 2703 may include mathematical libraries, deep learning libraries, parallel algorithm libraries, and / or signal / image / video processing libraries, which parallel computing applications (such as, but not limited to, application 2701) can utilize. CUDA library 2703 may include mathematical libraries, such as, but not limited to, the cuBLAS library (an implementation of the basic linear algebra subroutines (“BLAS”) for performing linear algebra operations), the cuFFT library (for computing the Fast Fourier Transform (“FFT”)), and the cuRAND library (for generating random numbers), etc. CUDA library 2703 may include deep learning libraries, such as, but not limited to, the cuDNN primitive library for deep neural networks and the TensorRT platform for high-performance deep learning inference, etc.

[0268] In at least one embodiment, the processor described elsewhere herein (e.g., but not limited to the one shown in Figure 1) Figure 14-26BThe processor (in the document) may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2700) to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0269] According to at least one embodiment, Figure 27 The software stack 2700 can execute in the ROCm implementation. Application 2701 can be launched on the ROCm software stack 2700, which includes a language runtime 2703, a system runtime 2705, a thunk 2707, and a ROCm kernel driver 2708. The ROCm software stack 2700 executes on hardware 2709, which may include a ROCm-enabled GPU developed by AMD Inc. of Santa Clara, California.

[0270] Application 2701 is executable and can be combined with the above. Figure 27 Similar functions are discussed. Furthermore, the language runtime 2703 and system runtime 2705 can execute functions related to the above. Figure 27 The discussed runtime 2705 has similar functionality. The difference between the language runtime 2703 and the system runtime 2705 is that the system runtime 2705 is a language-independent runtime that implements the ROCr System Runtime API 2704 and uses the Heterogeneous System Architecture (“HSA”) runtime API. The HSA runtime API may include a streamlined user-mode API that exposes interfaces for accessing and interacting with the AMD GPU, including functions for memory management, execution control via kernel architecture dispatch, error handling, system and agent information, and runtime initialization and shutdown. Unlike the system runtime 2705, the language runtime 2703 may be an implementation of the language-specific runtime API 2702, which sits in a layer above the ROCr System Runtime API 2704. The language runtime API may include the Heterogeneous Computing Portable Interface (“HIP”) language runtime API, the Heterogeneous Computing Compiler (“HCC”) language runtime API, or the OpenCL API, etc. HIP is an extension of the C++ programming language, offering a functionally similar version to the CUDA mechanism. Furthermore, the HIP runtime API can include features related to the above. Figure 27 The discussion covers functions similar to those in the CUDA runtime API, such as, but not limited to, memory management, execution control, device management, error handling, and synchronization.

[0271] The Thunk (ROCt) 2707 can be interface 2706, which is used to interact with the underlying ROCm driver 2708. The ROCm driver 2708 can be the ROCk driver, a combination of the AMD GPU driver and the HSA core driver (amdkfd). The AMD GPU driver can be a device core driver developed by AMD for GPUs, performing functions similar to those described above. Figure 27 The device kernel driver discussed is similar to the 2709. An HSA kernel driver can be a driver that allows different types of processors to share system resources more efficiently through hardware features.

[0272] Various libraries (not shown) may be included in the ROCm software stack 2700 on top of the language runtime 2703, and provide integration with the above. Figure 27 The discussion focuses on functionality similar to CUDA library 2703. Various libraries can include mathematical libraries, deep learning libraries, and / or other libraries, such as, but not limited to, the hipBLAS library which implements functionality similar to CUDA cuBLAS, the rocFFT library for computing FFTs similar to CUDA cuFFT, etc.

[0273] Processors described elsewhere in this document (e.g., but not limited to) Figure 14-26B The processor (in the document) may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2700) to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0274] According to at least one embodiment, Figure 27 The software stack 2700 can execute in an OpenCL implementation. The OpenCL software stack 2700, on which the application 2701 can be launched, may include the OpenCL framework 2703, the OpenCL runtime 2705, and a driver 2708. The OpenCL software stack 2700 can execute on non-vendor-specific hardware 2709. Because devices developed by different vendors support OpenCL, specific OpenCL drivers may be required for interoperability with the hardware of those vendors.

[0275] Application 2701, OpenCL runtime 2705, device kernel driver 2708, and hardware 2709 can execute in combination with the above. Figure 27Other implementations of the discussed application 2701, runtime 2705, device kernel driver 2708, and hardware 2709 offer similar functionality. Application 2701 may also include an OpenCL kernel (not shown), whose code will execute on the device.

[0276] OpenCL can define a "platform" that allows a host to control devices connected to it. The OpenCL framework provides platform-level APIs and runtime APIs, shown as Platform API 2702 and Runtime API 2704, respectively. Runtime API 2704 uses contexts to manage kernel execution on devices. Each identified device can be associated with a corresponding context, which Runtime API 2704 uses to manage the device's command queue, program objects, kernel objects, shared memory objects, and so on. Platform API 2702 exposes functions that allow the use of device contexts to select and initialize devices, submit work to devices via command queues, and enable data transfer with devices. In addition, the OpenCL framework provides various built-in functions (not shown), including mathematical functions, relational functions, and image processing functions.

[0277] The OpenCL framework 2703 may also include a compiler (not shown). Source code can be compiled offline before application execution or online during application execution. Unlike CUDA and ROCm, OpenCL applications can be compiled online by a compiler, which represents any number of compilers that can be used to compile source code and / or IR code (e.g., but not limited to Standard Portable Intermediate Representation (“SPIR-V”) code) into binary code. Alternatively, OpenCL applications can be compiled offline before execution.

[0278] In at least one embodiment, the processor described elsewhere herein (e.g., but not limited to) Figure 14-26B The processor (in the document) may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (e.g., software stack 2700) to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0279] According to at least one embodiment, the software may be supported by a programming platform configured to support various programming models, middleware and / or libraries, and frameworks that the application may rely on. The application may be an AI / ML application implemented using, for example, deep learning frameworks (such as, but not limited to, MXNet, PyTorch, or TensorFlow), which may rely on libraries such as, but not limited to, cuDNN, the NVIDIA Collective Communication Library (“NCCL”), and / or the NVIDIA Developer Data Loading Library (“DALI”) CUDA library to provide accelerated computation on the underlying hardware.

[0280] The programming platform can be a combination of the above. Figure 27 The platform described is one of CUDA, ROCm, or OpenCL. The programming platform can support various programming models, which can be abstractions of the underlying computing system that allow the expression of algorithms and data structures. Programming models can expose features of the underlying hardware to improve performance. Programming models may include CUDA, HIP, OpenCL, C++ Accelerated Massive Parallelism (“C++AMP”), Open Multiprocessing (“OpenMP”), Open Accelerators (“OpenACC”), and / or Vulkan Compute.

[0281] Libraries and / or middleware can provide abstract implementations of programming models. Such libraries may include data and programming code that computer programs can use and leverage during software development. Such middleware may include software that provides services to applications beyond those offered by the programming platform. Libraries and / or middleware may include cuBLAS, cuFFT, cuRAND, and other CUDA libraries, or rocBLAS, rocFFT, rocRAND, and other ROCm libraries. Furthermore, libraries and / or middleware may include the NCCL and ROCm communication collection library (“RCCL”) libraries that provide communication routines for GPUs, the MIOpen library for accelerating deep learning, and / or the Eigen library for linear algebra, matrix and vector operations, geometric transformations, numerical solvers, and related algorithms.

[0282] Application frameworks may depend on libraries and / or middleware. Each application framework can be a software framework that provides a standard structure for implementing application software. Returning to the AI / ML example discussed above, AI / ML applications can be implemented using frameworks such as, but not limited to, deep learning frameworks like Caffe, Caffe2, TensorFlow, Keras, PyTorch, or MxNet.

[0283] In at least one embodiment, the processor described elsewhere herein (e.g., but not limited to) Figure 14-26BThe processor (in the document) may include one or more circuits for executing an application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software (such as the programming platform described herein) to execute the application programming interface (API) to suspend and / or cause one or more indicated instructions or otherwise perform any of the operations described above or elsewhere herein.

[0284] Figure 28 The method for using at least one embodiment in the above is shown. Figure 27 Compiled code executed on one of the programming platforms shown. Compiler 2801 is configured to receive source code 2800, compile source code 2800, and output executable file 2810. Compiler 2801 can be configured to convert source code 2800 into host executable code 2807 for execution on a host and device executable code 2808 for execution on a device. Source code 2800 can be compiled offline before executing the application or can be compiled online during application execution. Source code 2800 can include code in any programming language supported by compiler 2801, such as, but not limited to, C++, C, Fortran, etc. Source code 2800 can be included in a single source file containing both host code and device code, with the location of the device code indicated. The single source file can be a .cu file including CUDA code, or a .hip.cpp file including HIP code, or a file in other formats including both host code and device code. Alternatively, source code 2800 can include multiple source code files instead of a single source file, with the host code and device code separated into these files. Compiler 2801 includes or can access one or more libraries to identify API call sequences to execute a single fused API, wherein the single fused API is a combination of two or more APIs. In at least one embodiment, compiler 2801 may be an NVIDIA CUDA compiler (“NVCC”) for compiling CUDA code in .cu files, or an HCC compiler for compiling HIP code in .hip.cpp files, or other compilers.

[0285] Compiler 2801 can be configured to compile source code 2800 into host executable code 2807 for execution on a host and device executable code 2808 for execution on a device. The operations performed by compiler 2801 include parsing source code 2800 into an abstract system tree (AST), performing optimizations, and generating executable code. When source code 2800 comprises a single source file, compiler 2801 can separate the device code from the host code in that single source file, compile the device code and host code into device executable code 2808 and host executable code 2807 respectively, and link the device executable code 2808 and host executable code 2807 together to form a single file.

[0286] Compiler 2801 may include compiler front-end 2802, host compiler 2805, device compiler 2806, and linker 2809. Compiler front-end 2802 may be configured to separate device code 2804 from host code 2803 in source code 2800. In at least one embodiment, device code 2804 may be compiled by device compiler 2806 into device executable code 2808, which may include binary code or IR code as described above. Separately, host code 2803 may be compiled by host compiler 2805 into host executable code 2807. For other compilers such as NVCC (e.g., but not limited to oneAPI, ROCm, and OpenCL compilers), host compiler 2805 may be a general-purpose C / C++ compiler that outputs native object code, while device compiler 2806 may be a low-level virtual machine (“LLVM”) based compiler that forks the LLVM compiler infrastructure and outputs PTX code or binary code. For HCC, both host compiler 2805 and device compiler 2806 may be LLVM based compilers that output object binary code.

[0287] After compiling source code 2800 into host executable code 2807 and device executable code 2808, linker 2809 can link host executable code 2807 and device executable code 2808 together to form executable file 2810. The host's native object code and the device's PTX or binary code can be linked together in an executable and linkable format (“ELF”) file, a container format for storing object code. Host executable code 2807 and device executable code 2808 can take any suitable format, such as, but not limited to, bin...

Claims

1. One or more processors, comprising: A circuit system for executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API.

2. The processor of claim 1 or more, wherein the one or more identifiers correspond to one or more contexts for controlling the GPU.

3. One or more processors according to claim 1, wherein the GPU instructions scheduled to be executed are associated with a first context for controlling the GPU, and wherein while the instructions associated with the first context are suspended, other instructions associated with a second context are executed.

4. The processor of claim 1, wherein the circuitry is configured to suspend the execution of the one or more GPU instructions by causing one or more schedulers to suspend the scheduling of instructions associated with the one or more identifiers.

5. One or more processors according to claim 1, wherein executing the API modifies the time slice allocation that the GPU will use to execute the instructions.

6. One or more processors according to claim 1, wherein executing the API causes the one or more GPU instructions to be removed from the GPU instruction queue to be executed by the GPU.

7. The processor of claim 1 or more, wherein the circuitry is configured to execute a second API to resume the execution of the one or more GPU instructions.

8. A system comprising: A circuit system for executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are associated with one or more identifiers indicated to the API.

9. The system of claim 8, wherein the one or more identifiers correspond to one or more contexts for controlling the GPU.

10. The system of claim 8, wherein the GPU instructions scheduled to be executed are associated with a first context for controlling the GPU, and wherein while the instructions associated with the first context are suspended, other instructions associated with a second context are executed.

11. The system of claim 8, wherein the circuitry is configured to suspend the execution of the one or more GPU instructions by causing one or more schedulers to suspend the scheduling of instructions associated with the one or more identifiers.

12. The system of claim 8, wherein executing the API modifies the time slice allocation of the GPU for executing the instructions.

13. The system of claim 8, wherein executing the API causes the one or more GPU instructions to be removed from the GPU instruction queue to be executed by the GPU.

14. The system of claim 8, wherein the circuitry is configured to execute a second API to resume the execution of the one or more GPU instructions.

15. A method comprising: Receive one or more application programming interface (API) parameters, which indicate the context for executing GPU instructions on the graphics processing unit (GPU); Identify one or more instructions that are scheduled to be executed on the GPU and are associated with the indicated context; as well as In response to receiving the parameter, the execution of the one or more instructions is suspended.

16. The method of claim 15, further comprising: This allows other instructions associated with the second context to be executed on the GPU while the instructions associated with the first context are paused.

17. The method of claim 15, wherein the execution of the one or more GPU instructions is suspended at least by causing one or more schedulers to suspend the scheduling of instructions associated with the one or more identifiers.

18. The method of claim 15, wherein the API modifies the allocation of time slices for the GPU to execute the one or more instructions.

19. The method of claim 15, wherein the context indicates a set of instructions, each of which is associated with a task to be executed by the GPU.

20. The method of claim 15, further comprising: Execute the second API to resume the execution of the one or more instructions.

21. One or more processors, comprising: A circuit system for executing an application programming interface (API) to suspend the execution of one or more GPU instructions scheduled to be executed on a graphics processing unit (GPU), wherein the GPU instructions are identified by the API as being associated with a program that has invoked the API.

22. The processor of claim 21, wherein the one or more GPU instructions are associated with a plurality of GPU contexts.

23. One or more processors according to claim 21, wherein the circuitry is configured to cause a scheduler to suspend scheduling instructions associated with the program.

24. One or more processors according to claim 21, wherein the circuitry is configured to execute one or more second instructions of a second program while instructions associated with the first program are suspended.

25. One or more processors according to claim 21, wherein the execution of the API modification is relative to the time slice in which the GPU will execute instructions associated with the second program, and the time slice in which the GPU will execute instructions associated with the program.

26. One or more processors of claim 21, wherein executing the API causes the one or more instructions to be removed from the instruction queue to be executed by the GPU.

27. One or more processors according to claim 21, wherein the circuitry is configured to execute a second API to resume the execution of the one or more instructions.

Images