System and method for allocating tasks in a neural network processor

Abstract

To manage tasks that, when executed by a neural processor circuit, instantiates a neural network.SOLUTION: The neural processor circuit includes neural engine circuits and a neural task manager circuit. The neural task manager circuit includes multiple task queues and a task arbiter circuit. Each task queue stores a reference to a task list of tasks for a machine learning operation. Each task queue may be associated with a priority parameter. Based on the priority of the task queues, the task arbiter circuit retrieves configuration data for a task from a memory external to the neural processor circuit, and provides the configuration data to components of the neural processor circuit including the neural engine circuits. The configuration data programs the neural processor circuit to execute the task.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] 1. Disclosure of technical field

[0002] The present disclosure relates to circuits for implementing neural networks, and more particularly to managing neural network tasks. [Background technology]

[0003] 2. Description of Related Art

[0004] An artificial neural network (ANN) is a computing system or model that uses a collection of connected nodes to process input data. ANNs are typically organized into layers, with different layers performing different types of transformations on their inputs. Extensions or variations of ANNs, such as convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs), have received significant attention. These computing systems or models often involve extensive computational operations, including multiplication and accumulation. For example, CNNs are a class of machine learning techniques that primarily use convolutions between input data and kernel data, which can be decomposed into multiplication and accumulation operations.

[0005] Depending on the type of input data and the operations performed, these machine learning systems or models may have different configurations. Such various configurations may include, for example, preprocessing operations, the number of channels in the input data, the kernel data used, nonlinear functions applied to the convolution results, and the application of various post-processing operations. It is relatively easy to instantiate and execute machine learning systems or models with various configurations using a central processing unit (CPU) and its main memory, because such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for various operations of these machine learning systems or models significantly consumes central processing unit (CPU) bandwidth and increases overall power consumption. Summary of the Invention

[0006] An embodiment relates to managing tasks that instantiate neural networks when executed by a neural processor circuit. The neural processor circuit includes one or more neural engine circuits and a neural task manager circuit. The neural task manager circuit includes a plurality of task queue circuits and a task arbiter circuit. Each task queue circuit stores a reference to a task list of tasks for machine learning operations. Each of the task lists and task queue circuits may be associated with a priority parameter. Based on the priority of the task queue circuits, the task arbiter circuit retrieves configuration data for the tasks from an external memory and provides the configuration data to components of the neural processor circuit, including one or more neural engines. The configuration data programs the neural processor circuit to perform the tasks.

[0007] Some embodiments include a method for managing tasks in a neural processor circuit. A reference to a task list of tasks for instantiating neural networks is stored in a task queue circuit of the neural processor circuit. The task list includes configuration data for the tasks stored in an external memory location. The task list is retrieved from the task queue circuit, e.g., based on task priority. In response to retrieving the task list, configuration data is retrieved from the external memory location. At least a portion of the configuration data is provided to a neural engine of the neural processor circuit to program the neural engine to perform the task. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of an electronic device according to one embodiment. [Figure 2] FIG. 1 is a block diagram illustrating components of an electronic device, according to one embodiment. [Figure 3] FIG. 2 is a block diagram illustrating a neural processor circuit, according to one embodiment. [Figure 4] FIG. 2 is a block diagram of a neural engine within a neural processor circuit, according to one embodiment. [Figure 5] FIG. 1 is a conceptual diagram illustrating a loop for processing input data in a neural processor circuit, according to one embodiment. [Figure 6] FIG. 1 is a conceptual diagram illustrating segmenting input data into slices, tiles, and work units according to one embodiment. [Figure 7] FIG. 1 illustrates programming of a rasterizer within a component of a neural processor circuit, according to one embodiment. [Figure 8] 1 is a flowchart illustrating a method for processing input data in a neural processor circuit, according to one embodiment. [Figure 9] FIG. 1 is a schematic block diagram illustrating a neural network represented by a list of tasks, according to one embodiment. [Figure 10]FIG. 2 is a block diagram illustrating a neural task manager, according to one embodiment. [Figure 11] FIG. 1 illustrates obtaining a task descriptor using a task queue, according to one embodiment. [Figure 12] FIG. 1 illustrates a task descriptor according to one embodiment. [Figure 13] FIG. 1 is a block diagram illustrating a fetch queue and a configuration queue for a neural task manager, according to one embodiment. [Figure 14] 1 is a flowchart illustrating a method for managing tasks in a neural processor circuit, according to one embodiment.

[0009] Various non-limiting embodiments are shown in the drawings and described in the detailed description for purposes of example only. DETAILED DESCRIPTION OF THE INVENTION

[0010] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various embodiments being described. However, the described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0011] Embodiments of the present disclosure relate to managing tasks that, when executed by a neural processor circuit, instantiate a neural network. Machine learning operations, such as inference or training operations, are defined by a task list of tasks. The neural processor circuit includes one or more neural engines and a neural task manager. The neural task manager includes multiple task queues and a task arbiter. Each task queue stores a task list of tasks for machine learning operations. Each task list or task queue may be associated with a priority parameter. The task arbiter retrieves configuration data for the tasks from an external memory based on the priority parameters and provides the configuration data to components of the neural processor circuit, including one or more neural engines. In some embodiments, the neural task manager includes a configuration queue that stores configuration data for committed tasks selected by the task arbiter and provides the configuration data to other components of the neural processor circuit. The configuration data programs the neural processor circuit to execute the tasks. For example, the configuration data may include input data and kernel data to be processed by the neural engines to execute the tasks. The configuration data may further include instructions for obtaining and processing the configuration data, as well as instructions for storing output data of the neural engine. Among other things, the neural task manager enables the neural processor circuit to efficiently process multiple machine learning operations. Furthermore, the neural task manager can facilitate task switching when a higher priority task is stored in a task queue while a lower priority task is being executed.

[0012] A "task" as used herein refers to a processing operation of a neural processor circuit that instantiates a network layer of a neural network, multiple network layers of a neural network, or a portion of a network layer of a neural network. A "task list" as used herein refers to a sequence of tasks, such as a sequence of tasks, performed by a neural processor circuit to instantiate multiple network layers of a neural network. Exemplary Electronic Devices

[0013] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as personal digital assistant (PDA) and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as wearable computers, laptop computers, or tablet computers, are optionally used. In some embodiments, the device is not a portable communication device, but is a desktop computer or other computing device not designed for portable use. In some embodiments, electronic devices of the present disclosure can include a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad). An exemplary electronic device (e.g., device 100) described below in connection with FIG. 1 can include a touch-sensitive surface for receiving user input. The electronic device can also include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0014] FIG. 1 is a schematic diagram of an electronic device 100 according to one embodiment. Device 100 may include one or more physical buttons, such as a "home" or menu button 104. Menu button 104 is used, for example, to navigate to any application within a set of applications running on device 100. In some embodiments, menu button 104 includes a fingerprint sensor that identifies a fingerprint on menu button 104. The fingerprint sensor can be used to determine whether a finger on menu button 104 has a fingerprint that matches a stored fingerprint to unlock device 100. Alternatively, in some embodiments, menu button 104 is implemented as a soft key within a graphical user interface (GUI) displayed on a touchscreen.

[0015] In some embodiments, device 100 includes a touchscreen 150, a menu button 104, a pushbutton 106 for turning power to the device on / off and locking the device, a volume control button 108, a Subscriber Identity Module (SIM) card slot 110, a headset jack 112, and an external docking / charging port 124. Pushbutton 106 can be used to turn power on / off on the device by pressing and holding the button down for a predetermined time interval, to lock the device by pressing and releasing the button before the predetermined time has elapsed, and / or to unlock the device or begin the unlocking process. In an alternative embodiment, device 100 also accepts verbal input via microphone 113 to activate or deactivate some features. Device 100 includes various components, including, but not limited to, memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include components not shown in FIG. 1 .

[0016] Device 100 is merely one example of an electronic device, and device 100 may have more or fewer components than those listed above, and some of the components may be combined into one component or have a different configuration or arrangement. The various components of device 100 listed above may be embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits (ASICs).

[0017] FIG. 2 is a block diagram illustrating components of device 100, according to one embodiment. Device 100 can perform various operations, including image processing. For this and other purposes, device 100 can include, among other components, an image sensor 202, a system-on-a-chip (SOC) component 204, system memory 230, persistent storage (e.g., flash memory) 228, a motion (orientation) sensor 234, and a display 216. The components shown in FIG. 2 are merely exemplary. For example, device 100 may include other components (such as a speaker or microphone) not shown in FIG. 2. Additionally, some components (such as motion sensor 234) may be omitted from device 100.

[0018] The image sensor 202 is a component for capturing image data and may be embodied, for example, as a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, a camera, a video camera, or other device. The image sensor 202 generates raw image data that is sent to the SOC component 204 for further processing. In some embodiments, the image data processed by the SOC component 204 is displayed on the display 216, stored in the system memory 230, in persistent storage 228, or sent to a remote computing device via a network connection. The raw image data generated by the image sensor 202 may be a Bayer color kernel array (CFA) pattern (hereinafter also referred to as a “Bayer pattern”).

[0019] The motion sensor 234 is a component or set of components for sensing movement of the device 100. The motion sensor 234 can generate a sensor signal indicative of the orientation and / or acceleration of the device 100. The sensor signal is sent to the SOC component 204 for various actions, such as turning on the device 100 or rotating an image displayed on the display 216.

[0020] Display 216 is a component for displaying images such as those generated by SOC components 204. Display 216 may include, for example, a liquid crystal display (LCD) device or an organic light emitting diode (OLED) device. Based on data received from SOC components 204, display 216 may display various images, such as menus, selected operating parameters, images captured by image sensor 202 and processed by SOC components 204, and / or other information received from a user interface (not shown) of device 100.

[0021] System memory 230 is a component for storing instructions for execution by SOC components 204 and for storing data processed by SOC components 204. System memory 230 can be embodied as any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, system memory 230 may store pixel data or other image data or statistics in various formats.

[0022] Persistent storage 228 is a component for storing data in a non-volatile manner. Persistent storage 228 retains data even when power is not available. Persistent storage 228 may be embodied as read-only memory (ROM), flash memory, or other non-volatile random access memory devices.

[0023] The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. The SOC component 204 may include, among other subcomponents, an image signal processor (ISP) 206, a central processor unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and a bus 232 connecting these subcomponents. The SOC component 204 may include more or fewer subcomponents than those shown in FIG. 2 .

[0024] ISP 206 is hardware that performs various stages of an image processing pipeline. In some embodiments, ISP 206 can receive raw image data from image sensor 202 and process the raw image data into a format that can be used by other subcomponents of SOC component 204 or components of device 100. ISP 206 can perform various image manipulation operations, such as image transform operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transforms, as described in more detail below with reference to FIG. 3 .

[0025] CPU 208 may be embodied using any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. CPU 208 may be a general-purpose or embedded processor using any of a variety of ISAs, such as x86, PowerPC, SPARC, RISC, ARM, or MIPS instruction set architectures (ISAs), or any other suitable ISA. While a single CPU is shown in FIG. 2, SOC component 204 may include multiple CPUs. In a multiprocessor system, each of the CPUs may, but need not, commonly implement the same ISA.

[0026] The graphics processing unit (GPU) 220 is a graphics processing circuit for executing graphics data. For example, the GPU 220 can render objects (e.g., including pixel data for an entire frame) to be displayed in a frame buffer. The GPU 220 may include one or more graphics processors capable of running graphics software or hardware acceleration of certain graphics operations to perform some or all of the graphics operations.

[0027] The neural processor circuit 218 is a circuit that performs various machine learning operations based on calculations including multiplication, addition, and accumulation. Such calculations may be configured, for example, to perform convolutions of input data and kernel data. The neural processor circuit 218 is a configurable circuit that performs these operations in a fast and power-efficient manner, while freeing the CPU 208 from resource-intensive operations associated with neural network operations. The neural processor circuit 218 may receive input data from the sensor interface 212, the image signal processor 206, the system memory 230, or other sources, such as the network interface 210 or the GPU 220. The output of the neural processor circuit 218 may be provided to various components of the device 100, such as the image signal processor 206, the system memory 230, or the CPU 208, for various operations. The structure and operation of the neural processor circuit 218 are described in detail below with reference to FIG. 3.

[0028] Network interface 210 is a subcomponent that allows data to be exchanged between device 100 and other devices over one or more networks (e.g., carrier or agent devices). For example, video or other image data may be received from other devices over network interface 210 and stored in system memory 230 for subsequent processing and display (e.g., via a back-end interface to image signal processor 206 as described below with respect to FIG. 3). Networks may include, but are not limited to, local area networks (LANs) (e.g., Ethernet or enterprise networks) and wide area networks (WANs). Image data received over network interface 210 may be subjected to image processing by ISP 206.

[0029] The sensor interface 212 is a circuit for interfacing with the motion sensor 234. The sensor interface 212 receives sensor information from the motion sensor 234 and processes this sensor information to determine the orientation or movement of the device 100.

[0030] Display controller 214 is a circuit for transmitting image data to be displayed on display 216. Display controller 214 receives image data from ISP 206, CPU 208, graphics processor, or system memory 230 and processes the image data into a format suitable for display on display 216.

[0031] The memory controller 222 is a circuit for communicating with the system memory 230. The memory controller 222 can read data from the system memory 230 for processing by the ISP 206, the CPU 208, the GPU 220, or other subcomponents of the SOC component 204. The memory controller 222 can also write data received from various subcomponents of the SOC component 204 to the system memory 230.

[0032] The video encoder 224 is hardware, software, firmware, or a combination thereof for encoding video data into a format suitable for storage on persistent storage 128 or for passing the data to the network interface 210 for transmission over a network to another device.

[0033] In some embodiments, one or more subcomponents of SOC component 204, or the functionality of some of these subcomponents, may be performed by software components executing on ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, persistent storage 228, or another device that communicates with device 100 via network interface 210.

[0034] Image or video data may flow through various data paths within SOC component 204. In one embodiment, raw image data may be generated from image sensor 202, processed by ISP 206, and then transmitted to system memory 230 via bus 232 and memory controller 222. After the image data is stored in system memory 230, it may be accessed via bus 232 by video encoder 224 for encoding or by display 116 for display. Exemplary Neural Processor Circuit

[0035] Neural processor circuit 218 is a configurable circuit that performs neural network operations on input data based at least on kernel data 340. To this end, neural processor circuit 218 may include, among other components, a neural task manager 310, a plurality of neural engines 314A-314N (hereinafter collectively referred to as "neural engines 314" and individually as "neural engines 314"), a kernel direct memory access (DMA) 324, a data buffer 318, and a buffer DMA 320. Neural processor circuit 218 may include other components not shown in FIG. 3.

[0036] Each of the neural engines 314 performs computational operations for neural network operations in parallel. Depending on the operational load, the entire set of neural engines 314 may be operated, or only a subset of the neural engines 314 may be operated while the remaining neural engines 314 are placed in a power save mode to conserve power. Each of the neural engines 314 includes components for storing one or more kernels, performing multiply-accumulate operations, and post-processing to generate output data 328, as described in more detail below with reference to FIG. 4. One example of a neural network operation is a convolution operation.

[0037] The neural task manager 310 manages the overall operation of the neural processor circuit 218. The neural task manager 310 may receive task lists from a compiler executed by the CPU 208, store tasks in its task queue, select tasks to execute, and send instructions to other components of the neural processor circuit 218 to execute the selected tasks. The neural task manager 310 may also perform task switching upon detecting an event, such as receiving an instruction from the CPU 208. In one or more embodiments, as described in more detail below with reference to FIGS. 5-7, the neural task manager 310 sends rasterizer information to components of the neural processor circuit 218 to enable each component to track, acquire, or process the appropriate portions of input data and kernel data. While the neural task manager 310 is shown in FIG. 3 as part of the neural processor circuit 218, the neural task manager 310 may be a component external to the neural processor circuit 218.

[0038] Kernel DMA 324 is a read circuit that fetches kernel data from a source (e.g., system memory 230) and sends kernel data 326A-326N to each of the neural engines 314. Kernel data represents information from which kernel elements may be extracted. In one embodiment, the kernel data may be in a compressed format that is compressed in each of the neural engines 314. While the kernel data provided to each of the neural engines 314 may be the same in some cases, the kernel data provided to each of the neural engines 314 will most likely be different.

[0039] The data buffer 318 is a temporary storage device for storing data associated with neural network operations. In one embodiment, the data buffer 318 is embodied as memory accessible by all of the neural engines 314. The data buffer 318 may store input data 322A-322N for supply to the corresponding neural engines 314A-314N, as well as outputs from each of the neural engines 314A-314N for feedback to the neural engines 314 or transmission to a target circuit (e.g., system memory 230). The operation of the data buffer 318 and other components of the neural processor circuit 218 is coordinated so that input data and intermediate data stored in the data buffer 318 are reused across multiple operations in the neural engines 314, thereby reducing data transfers to and from the system memory 230. The data buffer 318 may be operated in a broadcast mode, in which input data for all input channels is supplied to all neural engines 314, or in a unicast mode, in which input data for a subset of the input channels is supplied to each neural engine 314.

[0040] The input data 322 stored in the data buffer 318 may be, among other things, image data from a previous cycle of the neural engine 314, histogram of brightness gradient (HOG) data, audio data, metadata, output data 328, and portions of other processed data received from other components of the SOC component 204.

[0041] The buffer DMA 320 includes a read circuit that receives a portion (e.g., a tile) of input data from a source (e.g., system memory 230) for storage in the data buffer 318, and a write circuit that delivers the data from the data buffer 318 to a target (e.g., system memory). Exemplary Neural Engine Architecture

[0042] 4 is a block diagram of a neural engine 314 according to one embodiment. The neural engine 314 performs various operations to facilitate neural network operation, such as convolution, spatial pooling, and local response normalization. The neural engine 314 receives input data 322, performs multiply-accumulate operations (e.g., convolution operations) on the input data 322 based on stored kernel data, performs further post-processing operations on the results of the multiply-accumulate operations, and generates output data 328. The input data 322 and / or output data 328 of the neural engine 314 may be single-channel or multi-channel.

[0043] The neural engine 314 may include, among other components, an input buffer circuit 402, a computational core 416, a neural engine (NE) control unit 418, a kernel extraction circuit 432, an accumulator 414, and an output circuit 424. The neural engine 314 may include additional components not shown in FIG.

[0044] The input buffer circuit 402 receives input data 322 from the data buffer 318, stores portions of the input data, and transmits portions 408 of the input data appropriate for the current task or process loop to the computation core 416 for processing. The input buffer circuit 402 includes a shifter 410 that shifts the read location of the input buffer circuit 402 to modify the portions of the input data 408 that are transmitted to the computation core 416. By modifying the portions of the input data provided to the computation core 416 via shifting, the neural engine 314 can perform multiply-accumulate operations on different portions of the input data based on a fewer number of read operations. In one or more embodiments, the input data 322 includes data for difference convolution groups and / or input channels.

[0045] The kernel extraction circuit 432 is a circuit that receives the kernel data 326 from the kernel DMA 324 and extracts the kernel coefficients 422. In one embodiment, the kernel extraction circuit 432 references a lookup table (LUT) and uses a mask to reconstruct the kernel from the compressed kernel data 326. The mask indicates locations in the reconstructed kernel that are to be filled with zeros, and the remaining locations are filled with numbers. The kernel coefficients 422 of the reconstructed kernel are sent to the computation core 416 and loaded into registers in a multiply-add (MAD) circuit of the computation core 416. In another embodiment, the kernel extraction circuit 432 receives the kernel data in an uncompressed format, and the kernel coefficients are determined without reference to a LUT or using a mask.

[0046] Computation core 416 is a programmable circuit that performs computational operations. To this end, computation core 416 may include MAD circuits MAD0-MADN and a post-processor 428. Each of MAD circuits MAD0-MADN may store an input value in input data portion 408 and a corresponding kernel coefficient in kernel coefficients 422. The input value and the corresponding kernel coefficient are multiplied in each MAD circuit to generate processed value 412.

[0047] The accumulator 414 is a memory circuit that receives and stores the processed values ​​412 from the MAD circuit. The processed values ​​stored in the accumulator 414 may be sent as feedback information 419 for further multiplication and addition operations in the MAD circuit, or may be sent to a post-processor 428 for further processing. The accumulator 414, in combination with the MAD circuit, forms a multiply-accumulator (MAC) 404. In one or more embodiments, the accumulator 414 may have sub-units, each sending data to a different component of the neural engine 314. For example, during a processing cycle, data stored in a first sub-unit of the accumulator 414 is sent to the MAC circuit, and data stored in a second sub-unit of the accumulator 414 is sent to the post-processor 428.

[0048] The post-processor 428 is a circuit that performs further processing of the values ​​412 received from the accumulator 414. The post-processor 428 may perform operations including, but not limited to, non-linear functions (e.g., rectified linear unit (ReLU)), normalized cross-correlation (NCC), performing neural operations on 8-bit data and merging the results with 16-bit data, and local response normalization (LRN). The results of such operations are output from the post-processor 428 to the output circuit 424 as processed values ​​417.

[0049] The NE control unit 418 controls the operation of the other components of the neural engine 314 based on the operating mode and parameters of the neural processor circuit 218. Depending on different operating modes (e.g., grouped convolution mode or non-grouped convolution mode) or parameters (e.g., the number of input channels and the number of output channels), the neural engine 314 may operate in different sequences on different input data, return different values ​​from the accumulator 414 to the MAC circuit, and perform different types of post-processing operations in the post-processor 428. To configure the components of the neural engine 314 to operate in a desired manner, the NE control unit 418 sends control signals to the components of the neural engine. The NE control unit 418 may also include a rasterizer 430 that tracks the current task or process loop being processed by the neural engine 314, as described in more detail below with reference to FIGS. 5-7.

[0050] The output circuitry 424 receives the processed values ​​417 from the post processor 428 and interfaces with the data buffer 318 to store the processed values ​​417 in the data buffer 318. To this end, the output circuitry 424 may deliver the processed values ​​417 as output data 328 in sequence, out of sequence, or in the form in which they are processed in the post processor 428.

[0051] Components within the neural engine 314 may be configured during configuration by the NE control unit 418 and the neural task manager 310. To this end, the neural task manager 310 sends configuration information to the neural engine 314 during configuration. Configurable parameters and modes include, but are not limited to, the mapping between input data elements and kernel elements, the number of input channels, the number of output channels, the execution of the output stride, and enabling / selecting post-processing operations in the post processor 428. Operation of segmenting data for processing in a neural processor circuit

[0052] Input data is typically divided into smaller pieces for parallel processing on multiple neural engines 314. Often, multiple cycles of operation are performed to generate output for a task associated with a neural network. A compiler executed by CPU 208 analyzes the layers and nodes of the neural network and determines how the input data should be segmented based on the hardware constraints of the neural processor circuit 218. One of the compiler's functions is to determine how the input data should be divided into smaller data units for processing in the neural engines 314 and how the processing should be repeated in a loop to generate a result for the task.

[0053] 5 is a conceptual diagram illustrating a loop for processing input data in the neural processor circuit 218 according to one embodiment. The outermost loop represents processing for a convolution group when group convolution involving multiple convolution groups is used. A group convolution is a convolution in which input data for input channels within each group is used only to generate output data for the output channels of that group, but is not used to generate output data for output channels of other groups. Thus, each group of a group convolution can be treated as a separate convolution operation.

[0054] Each convolution group loop includes a processing loop for a slice of input data. The entire input data for the convolution operation is segmented into multiple overlapping strips of slices, as shown in FIG. 6. The overlapping portions 602, 604, and 606 are portions of the input data that are overfetched in two adjacent slices to provide spatial support for the corresponding kernel. The second outermost loop performs the convolution operation for each slice in the input data. Within the loop for a slice, there is a processing loop for tiles of the slice. Each slice is segmented into multiple tiles, as shown in FIG. 6. The overlapping portions 608, 610, 612, and 614 are portions of the input data in slice 4 that are overfetched in two adjacent tiles to provide spatial support for the corresponding kernel. The rightmost tile will typically have a smaller width than the other tiles in the slice. In one embodiment, the input data for each tile is loaded onto the data buffer 318 during a read cycle and reused for operations in the processing loop for the tile. Within the processing loop for a tile is a processing loop for a work unit. Each tile is segmented into multiple work units, as shown in FIG. 6. A work unit is a portion of input data sized to generate an output value that fits into the accumulator 414 of the neural engine 314 during a single cycle of the compute core 416. While the shape of each work unit is shown as a horizontal strip in FIG. 6, the shape of the work unit may vary depending on the shape and size of the tile. Work units also have overlapping portions that represent over-fetched data to provide support for the corresponding kernel. In particular, the work unit for the last tile of a slice may have the shape of a vertical strip if the tile is vertically oriented. In one or more embodiments, the size of each work unit is 256 bytes. In such embodiments, for example, the work units are shaped into one of the following dimensions: 16x16, 32x8, 64x4, 128x2, or 256x1.

[0055] For each work unit, an internal processing loop for an output channel group (OCG) may be provided. The number of output channels generated for a given work unit by a single cycle of the computational core 416 is referred to as an OCG. Depending on the mode of operation, each neural engine 314 may process output data for a different number of output channels (e.g., 8 channels, 32 channels) for a single load of input data into the input buffer circuit 402.

[0056] For each output channel group, an inner processing loop for the input channel (Cin) may be provided. If an input stride is implemented to skip certain input data, a loop for a sub-input channel (Sub-Cin) may be provided within the processing loop for the input channel (Cin).

[0057] For each input channel or sub-input channel, an inner loop is provided to process the horizontal spatial support for the kernel and the vertical support within each horizontal spatial support. The spatial support refers to the input data for convolution with the kernel and includes over-fetched input data at the end of the input data for performing the convolution.

[0058] Over-fetching refers to fetching additional input data within a current slice, tile, or work unit so that the appropriate dimensions of the input data can be provided for convolution with a kernel. In one or more embodiments, over-fetching is performed vertically between slices to obtain additional rows of input data (shown in FIG. 6 as overlapping portions 602, 604, 606), horizontally between tiles to obtain additional columns of input data (shown in FIG. 6 as overlapping portions 608, 610, 612, 614), and vertically between work units within a tile to obtain additional rows of input data.

[0059] For each spatial support for the kernel, an inner processing loop for an output channel (OC) is provided to generate output data for each output channel (Cout). If the output stride implements spatial upsampling, an additional inner loop is provided to process each sub-output channel. If the output stride is implemented, loading of kernel coefficients and MAC operations is performed within the loop for the output channel (OC) or sub-output channel to generate output data for the output channel (OC) or sub-output channel.

[0060] The nested loop structure of Figure 5 is merely illustrative. Loops may be omitted, added, or restructured depending on various factors. For example, if only a single convolution group is used, the outermost loop may be removed. Furthermore, the loop structures for horizontal and vertical spatial support may be reversed.

[0061] 5 and 6, operations associated with dividing the input space into smaller units and processing these smaller units are performed by rasterizers 714, 718, 720, 722 in various components of the neural processor circuit 218. Rasterizers are circuits in the various components of the neural processor circuit 218 that keep track of segments of input / output data (e.g., groups, work units, input channels, output channels) and instruct the components of the neural processor circuit for appropriate handling of the segments of input data. For example, rasterizer 720 in buffer DMA 320 keeps track of tiles and slices received from system memory 230, while rasterizer 718 in data buffer 318 broadcasts work units in order for processing by the neural engine 314. A rasterizer 722 in the kernel DMA 322 determines which kernels are received and distributed to the neural engine 314, while a rasterizer 714 in the neural engine 314 operates the shifter 410 in the input buffer circuit 402 to deliver the correct portions of input data 408 to the MAC 404 and send the final output data 328 to the data buffer 318.

[0062] 7 illustrates the programming of rasterizers 714, 718, 720, and 722 within components 314, 318, 320, and 322 of neural processor circuit 218, according to one embodiment. To perform their functions, each of rasterizers 714, 718, 720, and 722 receives task information 710 (e.g., configuration data) that indicates how input data and / or kernel data should be segmented and handled by each component of neural processor circuit 218. The task information includes information about the details of the current layer (e.g., dimensions of input and output data, dimensions of associated kernels, type of padding at the boundaries of input data). Rasterizers 714, 718, 720, and 722 may also receive constraints on their operation (e.g., whether to allow or disallow tile widths above a threshold).

[0063] Providing rasterizers within different components of the neural processor circuit 218 can reduce overhead in data communicated between components of the neural processor circuit 218. If a single central rasterizer were provided to control different components of the neural processor circuit 218, this data may require kernel data, input data, and output data to be communicated between components to identify associated positions within a task loop, such as convolution groups, tiles, slices, work units, input channels, and output channels. By using distributed rasterizers, separate metadata is not required to communicate kernel data, input data, and output data between components of the neural processor circuit 218. Example Process in Neural Engine Architecture

[0064] 8 is a flow chart illustrating one embodiment of a method for processing input data in neural processor circuit 218. The method may include different and / or additional steps, or the steps may be in a different order.

[0065] After neural task manager 310 programs rasterizers 714, 718, 720, 722, the process of operating buffer DMA 320 begins with rasterizer 720 instructing buffer DMA 320 (804) to receive tiles of input data from system memory 230. The tiles received by buffer DMA 320 are stored (806) in data buffer 318.

[0066] The rasterizer 718 in the data buffer 318 then instructs the data buffer 318 to send the work unit to one or more neural engines 314 (808). The work unit is then stored in the input buffer circuit 402 of the one or more neural engines 314.

[0067] In one or more embodiments, the input buffer circuit 402 selects (816) a portion of the work unit to send to the MAC 404 to perform the multiply-accumulate operation. The MAC 404 then performs (820) the multiply-accumulate operation on the selected portion of the work unit using a corresponding kernel. It then determines (824) whether the entire work unit has been processed in one or more neural engines 314. If the determination is no, the selected portion of the work unit is shifted (828) by the shifter 410, and the multiply-accumulate operation is performed again (820).

[0068] If it is determined 824 that the entire work unit has been processed, the process proceeds to determine whether all work units in the tile have been processed 832. If the determination is no, the process proceeds to the next work unit 836 by causing the data buffer 318 to send the next work unit to one or more neural engines 314 808, and the subsequent process repeats.

[0069] If it is determined (832) that all work units in the tile have been processed by the neural engine 314, the process proceeds to determine (840) whether all tiles of input data have been processed. If the determination is no, the process proceeds (844) to the next tile by having the rasterizer 720 instruct (804) the buffer DMA 320 to receive the next tile from the system memory 230, and the subsequent process repeats.

[0070] If it is determined that all tiles of the input data have been processed (840), the process ends for the current input data, and the process may then be repeated to process the next input data, or may proceed to the next task.

[0071] The process embodiment as described above with reference to Figure 8 is merely exemplary. Additional loops may be implemented as described above with reference to Figure 5. Furthermore, the sequence of processes may be modified or omitted. Exemplary Neural Task Manager Architecture

[0072] A neural network may include network layers or sub-layers that are instantiated or implemented as a series of tasks executed by the neural processor circuit 218. Figure 9 is a schematic block diagram illustrating a neural network 900 represented by a list of tasks 904, according to one embodiment. The neural network 900 includes convolutional layers C1, C2, and C3 (sub-layer C3 00 , C3 10 , C3 11 , C3 20 , and C3 21, C5, and C6, and network layers (or sub-layers) including pooling layers P2 and P4. Neural network 900 is one example of a neural network architecture that may be instantiated by neural processor circuit 218, i.e., when neural network 900 is converted into task list 904 and made executable by neural processor circuit 218. Other types of neural network architectures with different types of network layers or different orders of network layers may also be instantiated by neural processor circuit 218.

[0073] The neural network 900 is converted into a task list 904 by, for example, the CPU 208. The task list 904 includes task C1, task C2+P2, task C3, 00 +P4, Task C3 10 , Task C3 11 +P4, Task C3 20 , Task C3 21 +P4, Task C5 a , Task C5 b , and task C6. Each task is associated with a task descriptor that defines the configuration of neural processor circuit 218 for performing the task. Each task may correspond to a single network layer of neural network 900, a portion of a network layer of neural network 900, or multiple network layers of neural network 900. For example, task C1 corresponds to a single network layer C1, task C2+P2 corresponds to multiple network layers C2 and P2, task C5 a and C5 b each corresponds to a portion of network layer C5. The neural processor circuit 218 instantiates the neural network 900 by executing the tasks in the task list 904 under the control of the neural task manager 310.

[0074] 10 is a block diagram illustrating a neural task manager 310, according to one embodiment. The neural task manager 310 manages the execution of tasks for one or more neural networks 900 by the neural processor circuit 218. The neural task manager 310 may include, among other components, a task arbiter 1002, task queues 1004A-1004N (hereinafter collectively referred to as "task queues 1004" and individually as "task queue 1004"), a task manager direct memory access (DMA) 1006, a fetch queue 1008, and a configuration queue 1010. The neural task manager 310 may include other components not shown in FIG. 10.

[0075] The task arbiter 1002 is circuitry, or a combination of circuitry and firmware, executed by the neural processor circuit 218 that selects tasks from the task queue 1004. The task arbiter 1002 dequeues tasks from the task queue 1004 and places the tasks in the configuration queue 1010. While the task is in the configuration queue, the task is committed to execution and the neural processor circuit performs prefetching on input and kernel data before the task is executed by other components of the neural processor circuit 218. For example, the task arbiter 1002 may perform fixed-priority arbitration among multiple task queues 1004 and select the task with the highest priority from the task queue 1004 for acquisition of a task descriptor 1012 from the system memory 230 by the task manager DMA 1006.

[0076] The neural task manager 310 may include one or more task queues 1004. Each task queue 1004 is coupled to the CPU 208 and the task arbiter 1002. Each task queue 1004 receives a reference from the CPU 208 to a task list 904 of a task that, when executed by the neural processor circuit 218, instantiates the neural network 900. The reference stored in each task queue 1004 may include a set of pointers and counters that point to the task list 904 in a task descriptor 1012 in the system memory 230. Each task queue 1004 may further be associated with a priority parameter that defines the relative priority of the task queue 1004. The task descriptor 1012 of a task specifies the configuration of the neural processor circuit 218 for executing the task.

[0077] Task manager DMA 1006 is coupled to task arbiter 1002, system memory 230, and fetch queue 1008. Task manager DMA 1006 includes a read circuit that receives task descriptors 1012 for tasks from a source (e.g., system memory 230) for storage in fetch queue 1008. For example, task arbiter 1002 selects a task queue 1004 according to the priority of the task queue 1004 and controls task manager DMA 1006 to select task descriptors 1012 for the tasks using the task list 904 referenced by the selected task queue 1004.

[0078] The fetch queue 1008 is a single entry queue that stores task descriptors 1012 for tasks that are pending commitment to execute. The fetch queue 1008 is coupled to the task manager DMA 1006 to receive task descriptors 1012 from the system memory 230 and provide the task descriptors 1012 to the configuration queue 1010 or provide configuration data 1014 extracted from the task descriptors 1012 to the configuration queue 1010.

[0079] The configuration queue 1010 holds configuration data 1014 for multiple tasks that are committed to execution. If a task is in the configuration queue 1010, the kernel DMA 324 may fetch kernel data from the system memory 230 and store it in the kernel extraction circuit 432 of the neural engine 314, and the buffer DMA 320 may fetch input data from the system memory 230 and store it in the data buffer 318. To execute the task, the kernel extraction circuit 432 provides the prefetched kernel data to the MAC 404 of the neural engine 314, and the data buffer 318 provides the prefetched input data to the MAC 404 of the neural engine 314. In some embodiments, the configuration queue 1010 may include multiple queues that hold configuration data 1014 extracted from committed task descriptors 1012. As discussed in more detail in connection with FIG. 13 , the configuration queue 1010 is further coupled to other components of the neural processor circuit 218 to configure the neural processor circuit 218 according to the configuration data 1014.

[0080] 11 illustrates retrieval of a task descriptor 1012 using a task queue 1004, according to one embodiment. The task queue 1004 includes references, e.g., a set of pointers, to task descriptors 1012A-1012N stored in system memory 230. To that end, the task queue 1004 may include memory that stores a head parameter 1102, a network identifier (ID) 1104, a base address index 1106, a tail parameter 1108, a count parameter 1110, and a priority parameter 1112. The head parameter 1102 is a pointer to a location in system memory 230 that stores task descriptor 1012A, which is at the head of the task queue 1004. The network ID 1104 identifies the neural network 900 of the task descriptor 1012 at the head of the task queue 1004, and the base address index 1106 is an index into a base address table 1114 tagged with the network ID 1104 of the task descriptor 1012A at the head of the task queue 1004. The count parameter 1110 defines the number of task descriptors 1012 in the task queue 1004. The priority parameter 1112 defines the priority of the task queue 1004 and is used by the task arbiter 1002 to select among multiple task queues 1004.

[0081] Once a particular task queue 1004 is selected (e.g., in response to the priority parameter 1112), the task arbiter 1002 references the head parameter 1102, the network ID 1104, the base address index 1106, and the base address table 1114 to obtain a task descriptor 1012 from the system memory 230 and place the task descriptor 1012 in the fetch queue 1008 to initiate a commitment for task execution. At each configuration period, the task arbiter 1002 may continue to place task descriptors 1012 in the fetch queue 1008 in response to the order of tasks defined by the task list 904 of the task queue 1004, such as by obtaining the next task descriptor 1012B.

[0082] FIG. 12 illustrates a task descriptor 1012, according to one embodiment. The task arbiter 1002 places task descriptors 1012 from the system memory 230 into a fetch queue 1008, which then transfers the task descriptors 1012 to a configuration queue 1010. The task descriptor 1012 with the highest priority (e.g., entered first) in the configuration queue 1010 is used to configure the neural processor circuit 218 for execution during a configuration period. The task descriptor 1012 includes configuration data 1014, which includes a task descriptor header 1202 and address data 1204A-1204N (hereinafter referred to as "address data 1204"). The task descriptor header 1202 includes configuration data 1014 that configures various operations of the neural task manager 310, including operations related to task selection and task switching. For example, the task descriptor header 1202 may be parsed by the task arbiter 1002 to extract configuration data 1014 that programs the neural task manager 310 and other components of the neural processing circuit 218. The task descriptor header 1202 may include a task identifier (ID) 1206 that identifies the task, a neural network identifier (ID) 1208 that identifies the neural network 900 instantiated by the task, task switch parameters 1210 that define whether the neural task manager 310 should initiate a task switch after execution of the task (e.g., to execute a task in a different task queue 1004), input surface parameters 1212 that define whether input data for the task should be obtained from system memory 230 or data buffers 318, output surface parameters 1214 that define whether output data for the task should be stored in system memory 230 or data buffers 318, various (e.g., base address) pointers 1216 to facilitate programming of the neural processor circuit 218, and one or more debug / exception parameters 1218 that control event, exception, or debug logging.

[0083] Each instance of address data 1204A-1204N (collectively or individually referred to as "address data 1204") defines an address and data payload pair used to program components of the neural processor circuit 218. The data payload may include input data and kernel data used to execute a task. For example, each instance of address data 1204 includes register data defining the data payload, a register address defining a destination memory location in the neural processing circuit 218 to receive the register data, and a register count defining consecutive memory locations (e.g., registers) to be written with the register data. In some embodiments, the register address is combined with a base address stored in the header 1202 to define the full address of each memory location. If the task descriptors 1116 are generated at compile time, the actual runtime addresses may not be known. The base address table 1114 is used to avoid duplicating or updating every task descriptor with dynamically assigned addresses.

[0084] 13 is a block diagram illustrating a fetch queue 1008 and a configuration queue 1010, according to one embodiment. The configuration queue 1010 is coupled to the fetch queue 1008, which is coupled to system memory 230 via task manager DMA 1006. The configuration queue 1010 is further coupled to one or more of the rasterizers 714 of the neural engines 314, the rasterizers 718 of the data buffers 318, the rasterizers 720 of the buffer DMA 320, and the rasterizers 722 of the kernel DMA 322. The fetch queue 1008 stores task descriptors 1012 (e.g., including task descriptor headers 1202 and address data 1204A-1204N) for tasks that are pending but not committed to execution. The fetch queue 1008 reduces the latency of reading the next task descriptor 1012 from system memory 230 into the configuration queue 1010. The fetch queue 1008 stores the task descriptor 1012 with the highest priority, as determined by the task arbiter 1002. The task arbiter 1002 may replace a task descriptor 1012 stored in the fetch queue 1008 if a higher priority task descriptor 1012 is enqueued (e.g., from a higher priority task queue 1004). A task descriptor 1012 in the fetch queue 1008 does not initiate prefetching of input data or kernels and does not affect task queue priorities, pointers, or counters. Thus, a task descriptor 1012 in the fetch queue 1008 may be easily replaced by a higher priority task descriptor 1012 by writing the higher priority task descriptor 1012 to the fetch queue 1008. When a task descriptor 1012 stored in the configuration queue 1010 is executed by the neural processor circuit 218, the task descriptor 1012 stored in the fetch queue 1008 may be transferred to the configuration queue 1010 and another task descriptor 1012 for a subsequent task may be stored in the fetch queue 1008.

[0085] The configuration queue 1010 stores task descriptors 1012 for tasks committed for execution by the neural processor circuit 218. In some embodiments, the configuration queue 1010 includes multiple separate queues 1310, each of which stores a portion of the configuration data 1014 (including configuration data 1014A-1014E) extracted from the task descriptors 1012. Additionally, each queue 1310 is coupled to a corresponding component of the neural processor circuit 218 to program that component with the configuration data 1014. Through operation of the configuration queues 1010, the neural task manager 310 programs the rasterizers 714, 718, 720, and 722 to perform the functions described above in Figures 7 and 8. For example, queue 1310A is coupled to rasterizer 714 of neural engine 314 and provides configuration data 1014A that controls the operation of shifter 410 in input buffer circuit 402 to deliver the correct portions of input data 408 to MAC 404 and send final output data 328 to data buffer 318. Queue 1310B is coupled to rasterizer 718 of data buffer 318 and provides configuration data 1014B that controls the broadcast of input data (e.g., work units) by data buffer 318 for processing by neural engine 314. Queue 1310C is a read queue coupled to rasterizer 720 of buffer DMA 320 and provides configuration data 1014C that controls buffer DMA 320 to retrieve input data (e.g., tiles) from system memory 230 and store the input data in data buffer 318. Queue 1310D is a write queue coupled to rasterizer 720 of buffer DMA 320 and provides configuration data 1014D that controls buffer DMA 320 to store output data in system memory 230. Queue 1310E is coupled to rasterizer 722 of kernel DMA 322 and provides configuration data 1014E that controls which kernels are received and distributed to neural engine 314. In some embodiments, task descriptors 1012 or configuration data 1014 stored in configuration queue 1010 cannot be replaced and will be executed in first-in, first-out (FIFO) order. Example Processes in a Neural Task Manager Architecture

[0086] 14 is a flow chart illustrating, according to one embodiment, a method for managing tasks in neural processor circuit 218. The method may include different and / or additional steps, or the steps may be in a different order.

[0087] The CPU 208 generates 1402 a task list 904 of task descriptors 1012 for tasks that, when executed by the neural processor circuit 218, instantiate the neural network 900. For example, the CPU 208 may receive input data for a machine learning operation from the image sensor 202, the system memory 230, the persistent storage 228, the network interface 210, or some other component. The machine learning operation may include an inference operation or a training operation. The neural network 900 may include a neural network architecture including kernel data and network layers. The input data is applied to the neural network 900 to perform the machine learning operation. The kernel data and network layers of the neural network 900 may be computed during a machine learning training process. The CPU 208 performs a compilation operation (offline or on-the-fly) to turn the neural network description 900 into a linked list of task descriptors 1012, referred to herein as a task list 904. Each task is defined by a task descriptor 1012 and, when executed by the neural processor circuit 218, instantiates a single network layer, multiple network layers, or portions of network layers. Each task descriptor 1012 for a task includes configuration data 1014, such as a task descriptor header 1202, and address data 1204 that defines neural processor circuit 218 address and data payload pairs. The data payload may include kernel data and input data for the neural network 900. The configuration data 1014 further includes instructions that configure the operation of the rasterizers 714, 718, 720, and 722 to perform the task.

[0088] The CPU 208 stores 1404 the task list 904 of task descriptors 1012 in the system memory 230. In some embodiments, the CPU 208, or another CPU external to the electronic device 100, generates the task descriptors 1012 and stores the task descriptors 1012 in the persistent storage 228 or some other non-volatile memory. The task descriptors 1012 are loaded along with kernel data and input data in the system memory 230 for use by the neural processor circuit 218. The CPU 208 may be coupled to the system memory 230 via the bus 232 and the memory controller 222.

[0089] The task list 904 may include a set of pointers that reference the locations of the task descriptors 1012 in the system memory 230. Additionally, the CPU 208 may update parameters of the task descriptors 1102, such as memory addresses or network IDs. For example, the task list 904 may include a head parameter 1102 and a tail parameter 1108 that define the beginning and ending register addresses, respectively, of the system memory 230 where the task descriptors 1012 are stored. In some embodiments, the references to register addresses in the task list 904 may be partial addresses, and a base address table 1114 is used to define the full reference address into the system memory 230. In some embodiments, the CPU 208 may patch absolute addresses as needed. The CPU 208 may further configure the neural processor circuit 218 by setting its base address registers.

[0090] The task arbiter 1002 of the neural task manager 310 enqueues 1406 task lists 904 to be executed by the neural processor circuit 218. For example, the neural task manager 310 includes multiple task queues 1004A-1004N. Each task queue 1004 may store a reference to a task list 904. Furthermore, the task queues 1004 are prioritized for execution according to the priority parameters 1112 of the task lists 904 referenced by the task queue 1004.

[0091] In some embodiments, the CPU 208 executes the general configuration of the neural processor circuit 218 to perform the tasks. The CPU 208 may also start the neural processor circuit 218 if the neural processor circuit 218 is not already running.

[0092] The task arbiter 1002 selects 1408 the task queue with the highest priority for execution. For example, during each programming period, the task arbiter 1002 selects a task queue 1004 based on a comparison of the priority parameters of the task queues 1004 or a comparison of the task queues 1004 with the task lists 904, and executes tasks from the task lists 904 starting with the task queue 1004 with the highest priority.

[0093] The neural processor circuit 218 executes 1410 the task from the selected task queue. For example, the neural processor circuit 218 performs the method of FIG. 8 to execute the task. The configuration queue 1010 of the neural task manager 310 may provide configuration data 1014 (or task information 710) to the rasterizers 714, 718, 720, and 722 of the neural processor circuit 218 to program the neural processor circuit 218 to execute the task, as shown in FIG. 7. Furthermore, the execution of the task may include processing prefetched kernel data and input data while the task is in the configuration queue 1010. The execution of each task may include multiple processing loops to handle the input data, as shown in FIGS. 5 and 6. The task arbiter 1002 may dequeue the task from the configuration queue 1010 after the task has executed.

[0094] In some embodiments, to execute a task, the task arbiter 1002 controls the task manager DMA 1006 to obtain a task descriptor 1012 for a task in the task list 904 and store the task descriptor 1012 in the fetch queue 1008. After execution of a previously committed task, the task arbiter 1002 may dequeue the executed task by removing the task's task descriptor or configuration data from the configuration queue 1010. The task descriptor 1012 or extracted configuration data 1014 for the current task is then placed from the fetch queue 1008 into the configuration queue 1010. When the task is in the configuration queue 1010, the neural processor circuit 218 may initiate a prefetch operation by the kernel DMA 324 for kernel data from the system memory 230 to the kernel extraction circuit 432 and a prefetch operation by the buffer DMA 320 for input data from the system memory 230 to the data buffer 318.

[0095] If a different task queue has a higher priority, a task switch to the task queue with the higher priority may be performed if possible. To perform a task switch (e.g., to another machine learning operation), the task arbiter 1002 replaces a task in the fetch queue 1008 with another task descriptor referenced in the different task queue 1004. The task arbiter 1002 may resume the interrupted machine learning operation defined by the task list 904 after the task switch is complete by storing the replaced task in the fetch queue 1008.

[0096] In some embodiments, the CPU 208 determines 1408 a priority parameter 1110 for the task list 904 to select a task queue. The priority parameter 1110 defines the priority of the task of the machine learning operation relative to other tasks of other machine learning operations executed by the neural processor circuit 218. The CPU 208 may facilitate the execution of multiple machine learning operations. The machine learning operations may differ, such as by using different input data or different neural network architectures and kernel data. The CPU 208 may determine the priority parameter for the task list 904 programmatically or based on user input. The task descriptor 1012 referenced by the task queue 1004 with the highest priority may be selected for execution. The priority of a task queue may be determined by the CPU 208 or dynamically based on information from previously executed task descriptors.

[0097] The task arbiter 1002 determines 1412 whether all tasks in the task list 904 have been executed. If no tasks in the task list 904 have been executed, the process returns to step 1410, where the task arbiter 1002 executes any unexecuted tasks in the task list 904.

[0098] As each task in the task list 904 is executed, the task arbiter 1002 removes 1414 the task list 904 from the task queue 1004. In another embodiment, the reference to the task's task descriptor 1012 may be removed from the task queue 1004 after the task is executed. The process may then terminate. In another embodiment, the process may return to 1402, where the CPU 208 may continue to generate task descriptors for tasks for other machine learning operations. In that sense, the process may be repeated to place tasks in the task queue according to the specified task priority for execution of the machine learning operations by the neural processor circuit 218.

[0099] While particular embodiments and applications have been illustrated and described, it should be understood that the invention is not limited to the exact structure and components disclosed herein, and that various modifications, changes, and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope of the disclosure.

Claims

1. A neural engine circuit, a neural task manager circuit coupled to the neural engine circuit; 1. A neural processor circuit comprising: a task queue circuit configured to store a reference to a task list of task descriptors including tasks for instantiating a neural network, the task descriptors in the task list including configuration data for the tasks including a task descriptor header and address data defining a neural processor circuit address; a task arbiter circuit coupled to the task queue circuit; the task arbiter circuit includes: retrieving the task descriptor stored in a memory location external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; Selecting the task from the task list for execution; storing a task descriptor for a task in a fetch queue configured to store task descriptors for tasks that are pending and not committed for execution; transferring the task descriptor from the fetch queue to a configuration queue configured to store the dequeued task so that the dequeued task becomes a dequeued task committed for execution; providing a portion of the configuration data for the dequeued task to the neural engine circuitry configured to execute the dequeued task based on the portion of the configuration data; It is configured as follows: Neural processor circuit.

2. 2. The neural processor circuit of claim 1, wherein the dequeued tasks, when executed, instantiate a single network layer of the neural network, multiple network layers of the neural network, or a portion of a network layer of the neural network.

3. the task arbiter circuit is further configured to store the configuration data included in the task descriptor in the configuration queue of the neural task manager circuit, the configuration queue being coupled to the neural engine circuit and configured to provide the portion of the configuration data to the neural engine circuit; The neural processor circuit comprises: a kernel direct memory access (DMA) configured to retrieve kernel data for the dequeued task from the memory external to the neural processor circuit if the configuration data is stored in the configuration queue; a buffered direct memory access (DMA) configured to obtain input data for the dequeued tasks from the memory external to the neural processor circuit when the configuration data is stored in the configuration queue; 2. The neural processor circuit of claim 1 further comprising:

4. The neural task manager circuit includes: the fetch queue coupled to the configuration queue; a task manager direct memory access (DMA) coupled to the fetch queue and to the task arbiter circuit; Further comprising: the task arbiter circuit is further configured to obtain, via the task manager DMA, the configuration data of the dequeued task from the location of the memory external to the neural processor circuit, and store the configuration data in the fetch queue, the fetch queue providing the configuration data to the configuration queue when a second configuration data of an executed task is removed from the configuration queue.

4. The neural processor circuit of claim 3.

5. the task queue circuit is a first task queue circuit, the neural task manager circuit further includes a second task queue circuit, the first task queue circuit having a first priority and the second task queue circuit having a second priority; the task arbiter circuitry is configured to store one of: (i) the configuration data of the dequeued task; or (ii) another configuration data of another task in the fetch queue based on a comparison of the first priority and the second priority.

5. The neural processor circuit of claim 4.

6. 2. The neural processor circuit of claim 1, further comprising a data buffer coupled to the memory and to the neural engine circuit that is external to the neural processor circuit, the task arbiter circuit being further configured to provide another portion of the configuration data to the data buffer, the other portion of the configuration data programming the data buffer to broadcast work units of input data for the dequeued tasks to the neural engine circuit.

7. a buffer direct memory access (DMA) coupled to the data buffer and to the memory external to the neural processor circuit; the task arbiter circuit is further configured to provide a third portion of the configuration data to the buffer DMA, the third portion of the configuration data programming the buffer DMA to obtain tiles of the input data for the dequeued tasks from the memory external to the neural processor circuit and store the tiles in the data buffer; The neural processor circuit of claim 6 , wherein the tile comprises a plurality of work units.

8. The neural engine circuit comprises: an input buffer circuit coupled to the neural task manager circuit; a multiply-add (MAD) circuit coupled to the input buffer circuit; Including, the portion of the configuration data programs the input buffer circuit to provide the portion of the input data for the dequeued tasks stored in the input buffer circuit to the MAD circuit; 2. The neural processor circuit of claim 1.

9. 9. The neural processor circuit of claim 8, further comprising a data buffer coupled to said memory and to said neural engine circuit external to said neural processor circuit, said neural engine circuit further including an output circuit, said portion of said configuration data programming said output circuit to provide output data from said MAD circuit to said data buffer.

10. a kernel direct memory access (DMA) coupled to the memory external to the neural processor circuit and to the neural engine circuit; 2. The neural processor circuit of claim 1, wherein the task arbiter circuit is further configured to provide another portion of the configuration data to the kernel DMA, the another portion of the configuration data programming the kernel DMA to retrieve kernel data from the memory external to the neural processor circuit and provide the kernel data to the neural engine circuit to execute the dequeued tasks.

11. 1. A method of managing tasks in a neural processor circuit, comprising: Referencing a plurality of tasks in a plurality of task queue circuits; storing in a task queue circuit of said neural processor circuit a reference to a task list of task descriptors including tasks for instantiating neural networks, the task descriptors in said task list including configuration data for the tasks including a task descriptor header and address data defining the neural processor circuit address; in a task arbiter circuit coupled to the plurality of task queue circuits, obtaining the task descriptor stored in a memory location external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; selecting the task from the task list for execution; storing a task descriptor for a task in a fetch queue configured to store task descriptors for tasks that are pending and not committed for execution; transferring the task descriptor from the fetch queue to a configuration queue configured to store the dequeued task so that the task descriptor becomes a dequeued task that is committed for execution; providing a portion of the configuration data for the dequeued task to a neural engine circuit configured to execute the dequeued task based on the portion of the configuration data; The method includes:

12. 12. The method of claim 11, wherein the dequeued task, when executed, instantiates a single network layer of the neural network, multiple network layers of the neural network, or a portion of a network layer of the neural network.

13. storing said configuration data in said configuration queue of said neural processor circuit; providing said portion of said configuration data from said configuration queue to said neural engine circuit; obtaining kernel data for the dequeued task from an external memory if the configuration data is stored in the configuration queue; obtaining input data for the dequeued task from the external memory if the configuration data is stored in the configuration queue; The method of claim 11 further comprising:

14. Retrieving the configuration data for the dequeued tasks from the location in the external memory via a task manager direct memory access (DMA) of the neural processor circuit; storing said configuration data in said fetch queue of said neural processor circuit; providing configuration data from the fetch queue to the configuration queue when other configuration data for an executed task is removed from the configuration queue; The method of claim 13 further comprising:

15. storing a first priority corresponding to the first task queue circuit and a second priority corresponding to the second task queue circuit; storing one of: (i) the configuration data of the dequeued task; or (ii) other configuration data of another task in the fetch queue based on a comparison of the first priority and the second priority; 15. The method of claim 14 further comprising:

16. 12. The method of claim 11, further comprising providing another portion of the configuration to a data buffer of the neural processor circuit, the another portion of the configuration data programming the data buffer to broadcast work units of input data for the dequeued tasks to the neural engine circuit.

17. 17. The method of claim 16, further comprising providing a third portion of the configuration data to a buffer direct memory access (DMA) of the neural processor circuit coupled to the data buffer and an external memory, the third portion of the configuration data programming the buffer DMA to obtain tiles of the input data for the dequeued tasks from the external memory and store the tiles in the data buffer, the tiles comprising a plurality of work units.

18. the portion of the configuration data programs an input buffer circuit of the neural engine circuit to provide a portion of the input data stored in the input buffer circuit for the dequeued task to a multiply-add (MAD) circuit of the neural engine circuit; said portion of said configuration data programs an output circuit of said neural engine circuit to provide output data from said MAD circuit to a data buffer of said neural processor circuit; The method of claim 11.

19. providing another portion of the configuration data to a kernel direct memory access (DMA) coupled to the memory external to the neural processor circuit and to the neural engine circuit, the another portion of the configuration data programming the kernel DMA to retrieve kernel data from the memory external to the neural processor circuit and to provide the kernel data to the neural engine circuit to perform the task. The method of claim 11.

20. 1. An integrated circuit (IC) system comprising a neural processor circuit, the neural processor circuit comprising: A neural engine circuit, a neural task manager circuit coupled to the neural engine circuit; the neural task manager circuit comprises: a plurality of task queue circuits referencing a plurality of tasks, one task queue circuit being configured to store a reference to a task list of task descriptors including a task for instantiating a neural network, the task descriptors in the task list including configuration data for the task including a task descriptor header and address data defining a neural processor circuit address; a task arbiter circuit coupled to the plurality of task queue circuits; the task arbiter circuit includes: retrieving the task descriptor stored in a memory location external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; Selecting the task from the task list for execution; storing a task descriptor for a task in a fetch queue configured to store task descriptors for tasks that are pending and not committed for execution; transferring the task descriptor from the fetch queue to a configuration queue configured to store the dequeued task so that the dequeued task becomes a dequeued task committed for execution; providing a portion of the configuration data for the dequeued task to the neural engine circuitry configured to execute the dequeued task based on the portion of the configuration data; It is configured as follows: Integrated circuit (IC) systems.

Images