System and method for assigning task in neural network processor

Abstract

To provide a system and a method for enabling task management that instantiates a neural network in a neural processor circuit.SOLUTION: In a neural processor circuit including neural engine circuits and a neural task manager circuit, the neural task manager includes multiple task queues and a task arbiter. Each task queue stores a reference to a task list of tasks for a machine learning operation, and is associated with a priority parameter. On the basis of the priority of the task queues, the task arbiter retrieves configuration data 1014 for a task from a memory external to the neural processor circuit, and provides the configuration data to components of the neural processor circuit including the neural engine circuits. The configuration data programs the neural processor circuit to execute the task.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] 1. Disclosure of technical field

[0002] The present disclosure relates to circuits for implementing neural networks, and more particularly, to managing neural network tasks. [Background technology]

[0003] 2. Description of Related Art

[0004] An artificial neural network (ANN) is a computing system or model that uses a collection of connected nodes to process input data. ANNs are typically organized into layers, with different layers performing different types of transformations on their inputs. Extensions or variations of ANNs, such as convolution neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs), have received significant attention. These computing systems or models often involve extensive computational operations, including multiplication and accumulation. For example, CNNs are a class of machine learning techniques that primarily use convolutions between input data and kernel data, which can be decomposed into multiplication and accumulation operations.

[0005] Depending on the type of input data and the operations performed, these machine learning systems or models may have different configurations. Such various configurations may include, for example, pre-processing operations, the number of channels in the input data, the kernel data used, the non-linear functions applied to the convolution results, and the application of various post-processing operations. It is relatively easy to instantiate and execute machine learning systems or models with various configurations using a central processing unit (CPU) and its main memory, because such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for various operations of these machine learning systems or models will consume significant central processing unit (CPU) bandwidth as well as increase overall power consumption. Summary of the Invention

[0006] An embodiment relates to managing tasks that instantiate a neural network when executed by a neural processor circuit. The neural processor circuit includes one or more neural engine circuits and a neural task manager circuit. The neural task manager circuit includes a plurality of task queue circuits and a task arbiter circuit. Each task queue circuit stores a reference to a task list of tasks for machine learning operations. Each of the task lists and the task queue circuits may be associated with a priority parameter. Based on the priority of the task queue circuits, the task arbiter circuit retrieves configuration data for the tasks from an external memory and provides the configuration data to components of the neural processor circuit, including one or more neural engines. The configuration data programs the neural processor circuit to execute the tasks.

[0007] Some embodiments include a method of managing tasks in a neural processor circuit. A reference to a task list of tasks for instantiating a neural network is stored in a task queue circuit of the neural processor circuit. The task list includes configuration data for the tasks stored in an external memory location. The task list is retrieved from the task queue circuit, e.g., based on task priority. In response to retrieving the task list, the configuration data is retrieved from the external memory location. At least a portion of the configuration data is provided to a neural engine of the neural processor circuit to program the neural engine to execute the tasks. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an electronic device according to one embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating components of an electronic device according to one embodiment. [Diagram 3] FIG. 2 is a block diagram illustrating a neural processor circuit, according to one embodiment. [Figure 4] FIG. 2 is a block diagram of a neural engine within a neural processor circuit according to one embodiment. [Diagram 5] FIG. 2 is a conceptual diagram illustrating a loop for processing input data in a neural processor circuit, according to one embodiment. [Figure 6] FIG. 2 is a conceptual diagram illustrating segmenting input data into slices, tiles, and work units according to one embodiment. [Figure 7] FIG. 2 illustrates programming of a rasterizer within a component of a neural processor circuit, according to one embodiment. [Figure 8] 4 is a flow chart illustrating a method for processing input data in a neural processor circuit according to one embodiment. [Figure 9] FIG. 2 is a schematic block diagram illustrating a neural network represented by a list of tasks according to one embodiment. [Figure 10]FIG. 2 is a block diagram illustrating a neural task manager according to one embodiment. [Figure 11] FIG. 1 illustrates obtaining a task descriptor using a task queue according to one embodiment. [Figure 12] FIG. 2 illustrates a task descriptor according to one embodiment. [Figure 13] FIG. 2 is a block diagram illustrating a fetch queue and a configuration queue for a neural task manager according to one embodiment. [Figure 14] 1 is a flow chart illustrating a method for managing tasks in a neural processor circuit according to one embodiment.

[0009] Various non-limiting embodiments are shown in the drawings and described in the detailed description for purposes of illustration only. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments being described. However, the described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0011] Embodiments of the present disclosure relate to managing tasks that, when executed by a neural processor circuit, instantiate a neural network. A machine learning operation, such as an inference operation or a training operation, is defined by a task list of tasks. The neural processor circuit includes one or more neural engines and a neural task manager. The neural task manager includes a plurality of task queues and a task arbiter. Each task queue stores a task list of tasks for machine learning operations. Each of the task lists or task queues may be associated with a priority parameter. The task arbiter retrieves configuration data for the tasks from an external memory based on the priority parameters and provides the configuration data to components of the neural processor circuit, including the one or more neural engines. In some embodiments, the neural task manager includes a configuration queue that stores configuration data for committed tasks selected by the task arbiter and provides the configuration data to other components of the neural processor circuit. The configuration data programs the neural processor circuit to execute the tasks. For example, the configuration data may include input data and kernel data to be processed by the neural engines to execute the tasks. The configuration data may further include instructions for obtaining and processing the configuration data, and instructions for storing output data of the neural engine. Among other things, the neural task manager enables the neural processor circuit to efficiently process multiple machine learning operations. Additionally, the neural task manager can facilitate task switching when a higher priority task is stored in a task queue while a lower priority task is being executed.

[0012] A "task" as used herein refers to a processing operation of a neural processor circuit that instantiates a network layer of a neural network, multiple network layers of a neural network, or a portion of a network layer of a neural network. A "task list" as used herein refers to a sequence of tasks, such as a sequence of tasks executed by a neural processor circuit to instantiate multiple network layers of a neural network. Exemplary Electronic Devices

[0013] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a personal digital assistant (PDA) function and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Other portable electronic devices, such as wearable computers, laptop computers, or tablet computers, are optionally used. In some embodiments, the device is not a portable communication device, but is a desktop computer or other computing device not designed for portable use. In some embodiments, electronic devices of the present disclosure can include a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad). Exemplary electronic devices (e.g., device 100) described below in connection with FIG. 1 can include a touch-sensitive surface for receiving user input. The electronic device can also include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0014] FIG. 1 is a schematic diagram of an electronic device 100 according to one embodiment. The device 100 may include one or more physical buttons, such as a "home" or menu button 104. The menu button 104 is used, for example, to navigate to any application in a set of applications running on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that identifies a fingerprint on the menu button 104. The fingerprint sensor can be used to determine whether a finger on the menu button 104 has a fingerprint that matches a stored fingerprint to unlock the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touch screen.

[0015] In some embodiments, device 100 includes a touch screen 150, a menu button 104, a push button 106 for turning power to the device on / off and locking the device, a volume control button 108, a Subscriber Identity Module (SIM) card slot 110, a headset jack 112, and an external port 124 for docking / charging. Push button 106 may be used to turn power on / off on the device by pressing and holding the button down for a predefined time interval, to lock the device by pressing and releasing the button before the predefined time has elapsed, and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, device 100 also accepts verbal input via microphone 113 to activate or deactivate some features. Device 100 includes various components, including, but not limited to, memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, speaker 111, microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include components not shown in FIG.

[0016] Device 100 is just one example of an electronic device, and device 100 may have more or fewer components than those listed above, and some of the components may be combined into one component or have a different configuration or arrangement. The various components of device 100 listed above may be embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits (ASICs).

[0017] FIG. 2 is a block diagram illustrating components of device 100, according to one embodiment. Device 100 can perform various operations, including image processing. For this and other purposes, device 100 can include, among other components, an image sensor 202, a system-on-a-chip (SOC) component 204, a system memory 230, persistent storage (e.g., flash memory) 228, a motion (orientation) sensor 234, and a display 216. The components as shown in FIG. 2 are merely exemplary. For example, device 100 may include other components (such as a speaker or microphone) not shown in FIG. 2. Additionally, some components (such as motion sensor 234) may be omitted from device 100.

[0018] The image sensor 202 is a component for capturing image data and may be embodied as, for example, a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, a camera, a video camera, or other device. The image sensor 202 generates raw image data that is sent to the SOC component 204 for further processing. In some embodiments, the image data processed by the SOC component 204 is displayed on the display 216, stored in the system memory 230, persistent storage 228, or sent to a remote computing device via a network connection. The raw image data generated by the image sensor 202 may be a Bayer color kernel array (CFA) pattern (hereinafter also referred to as "Bayer pattern").

[0019] The motion sensor 234 is a component or set of components for sensing the movement of the device 100. The motion sensor 234 can generate a sensor signal indicative of the orientation and / or acceleration of the device 100. The sensor signal is sent to the SOC component 204 for various actions, such as turning on the device 100 or rotating an image displayed on the display 216.

[0020] The display 216 is a component for displaying images such as those generated by the SOC components 204. The display 216 may include, for example, a liquid crystal display (LCD) device or an organic light emitting diode (OLED) device. Based on the data received from the SOC components 204, the display 216 may display various images, such as menus, selected operating parameters, images captured by the image sensor 202 and processed by the SOC components 204, and / or other information received from a user interface (not shown) of the device 100.

[0021] System memory 230 is a component for storing instructions for execution by SOC components 204 and for storing data processed by SOC components 204. System memory 230 may be embodied as any type of memory including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, system memory 230 may store pixel data or other image data or statistics in various formats.

[0022] Persistent storage 228 is a component for storing data in a non-volatile manner. Persistent storage 228 retains data even when power is not available. Persistent storage 228 may be embodied as read-only memory (ROM), flash memory, or other non-volatile random access memory devices.

[0023] The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. The SOC component 204 may include, among other subcomponents, an image signal processor (ISP) 206, a central processor unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and a bus 232 connecting these subcomponents. The SOC component 204 may include more or fewer subcomponents than those shown in FIG. 2.

[0024] The ISP 206 is hardware that performs various stages of an image processing pipeline. In some embodiments, the ISP 206 can receive raw image data from the image sensor 202 and process the raw image data into a format that can be used by other subcomponents of the SOC component 204 or components of the device 100. The ISP 206 can perform various image manipulation operations, such as image transformation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as described in more detail below with reference to FIG.

[0025] CPU 208 may be embodied using any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. CPU 208 may be a general purpose or embedded processor using any of a variety of ISAs, such as x86, PowerPC, SPARC, RISC, ARM, or MIPS instruction set architectures (ISAs), or any other suitable ISA. Although a single CPU is shown in FIG. 2, SOC component 204 may include multiple CPUs. In a multiprocessor system, each of the CPUs may, but does not necessarily, commonly implement the same ISA.

[0026] Graphics processing unit (GPU) 220 is a graphics processing circuit for executing graphics data. For example, GPU 220 can render objects (e.g., including pixel data for an entire frame) to be displayed in a frame buffer. GPU 220 may include one or more graphics processors capable of running graphics software or hardware acceleration of certain graphics operations to perform some or all of the graphics operations.

[0027] The neural processor circuit 218 is a circuit that performs various machine learning operations based on calculations including multiplication, addition, and accumulation. Such calculations may be configured to perform, for example, convolution of input data and kernel data. The neural processor circuit 218 is a configurable circuit that performs these operations in a fast and power-efficient manner while relieving the CPU 208 from resource-intensive operations associated with neural network operations. The neural processor circuit 218 may receive input data from the sensor interface 212, the image signal processor 206, the system memory 230, or other sources such as the network interface 210 or the GPU 220. The output of the neural processor circuit 218 may be provided to various components of the device 100, such as the image signal processor 206, the system memory 230, or the CPU 208 for various operations. The structure and operation of the neural processor circuit 218 are described in detail below with reference to FIG. 3.

[0028] The network interface 210 is a subcomponent that allows data to be exchanged between the device 100 and other devices over one or more networks (e.g., carrier or agent devices). For example, video or other image data may be received from other devices over the network interface 210 and stored in the system memory 230 for subsequent processing and display (e.g., via a back-end interface to the image signal processor 206 as described below with respect to FIG. 3). Networks may include, but are not limited to, Local Area Networks (LANs) (e.g., Ethernet or enterprise networks) and Wide Area Networks (WANs). Image data received over the network interface 210 may be subjected to image processing by the ISP 206.

[0029] The sensor interface 212 is a circuit for interfacing with the motion sensor 234. The sensor interface 212 receives sensor information from the motion sensor 234 and processes the sensor information to determine the orientation or movement of the device 100.

[0030] The display controller 214 is a circuit for transmitting image data to be displayed on the display 216. The display controller 214 receives image data from the ISP 206, the CPU 208, the graphics processor, or the system memory 230, and processes the image data into a format suitable for display on the display 216.

[0031] The memory controller 222 is a circuit for communicating with the system memory 230. The memory controller 222 can read data from the system memory 230 for processing by the ISP 206, the CPU 208, the GPU 220, or other subcomponents of the SOC component 204. The memory controller 222 can also write data received from various subcomponents of the SOC component 204 to the system memory 230.

[0032] The video encoder 224 is hardware, software, firmware, or a combination thereof for encoding the video data into a format suitable for storage in persistent storage 128 or for passing the data to the network interface 210 for transmission over a network to another device.

[0033] In some embodiments, one or more subcomponents of the SOC component 204, or the functionality of some of these subcomponents, may be performed by software components executing on the ISP 206, the CPU 208, or the GPU 220. Such software components may be stored in the system memory 230, the persistent storage 228, or another device in communication with the device 100 via the network interface 210.

[0034] Image data or video data may flow through various data paths within the SOC component 204. In one embodiment, raw image data may be generated from the image sensor 202, processed by the ISP 206, and then transmitted to the system memory 230 via the bus 232 and the memory controller 222. After the image data is stored in the system memory 230, it may be accessed via the bus 232 by the video encoder 224 for encoding or by the display 116 for display. Exemplary Neural Processor Circuit

[0035] The neural processor circuit 218 is a configurable circuit that performs neural network operations on input data based at least on kernel data 340. To this end, the neural processor circuit 218 may include, among other components, a neural task manager 310, a plurality of neural engines 314A-314N (hereinafter collectively referred to as "neural engines 314" and individually referred to as "neural engines 314"), a kernel direct memory access (DMA) 324, a data buffer 318, and a buffered DMA 320. The neural processor circuit 218 may include other components not shown in FIG.

[0036] Each of the neural engines 314 performs computational operations for neural network operations in parallel. Depending on the operational load, the entire set of neural engines 314 may be operated, or only a subset of the neural engines 314 may be operated while the remaining neural engines 314 are placed in a power save mode to conserve power. Each of the neural engines 314 includes components for storing one or more kernels, for performing multiply-accumulate operations, and for post-processing to generate output data 328, as described in more detail below with reference to FIG. 4. One example of a neural network operation is a convolution operation.

[0037] The neural task manager 310 manages the overall operation of the neural processor circuit 218. The neural task manager 310 may receive task lists from a compiler executed by the CPU 208, store tasks in its task queue, select a task to execute, and send instructions to other components of the neural processor circuit 218 to execute the selected task. The neural task manager 310 may also perform task switching upon detecting an event, such as receiving an instruction from the CPU 208. In one or more embodiments, as described in more detail below with reference to FIGS. 5-7, the neural task manager 310 sends rasterizer information to the components of the neural processor circuit 218 to enable each component to track, acquire, or process the appropriate portions of input data and kernel data. Although the neural task manager 310 is shown in FIG. 3 as part of the neural processor circuit 218, the neural task manager 310 may be a component external to the neural processor circuit 218.

[0038] Kernel DMA 324 is a read circuit that fetches kernel data from a source (e.g., system memory 230) and transmits kernel data 326A-326N to each of the neural engines 314. The kernel data represents information from which kernel elements may be extracted. In one embodiment, the kernel data may be in a compressed format that is compressed in each of the neural engines 314. While the kernel data provided to each of the neural engines 314 may be the same in some cases, the kernel data provided to each of the neural engines 314 will most likely be different.

[0039] The data buffer 318 is a temporary storage device for storing data associated with neural network operations. In one embodiment, the data buffer 318 is embodied as a memory accessible by all of the neural engines 314. The data buffer 318 may store input data 322A-322N for feeding to the corresponding neural engines 314A-314N, as well as outputs from each of the neural engines 314A-314N for feeding back to the neural engines 314 or for transmission to a target circuit (e.g., the system memory 230). The operation of the data buffer 318 and other components of the neural processor circuit 218 is coordinated such that the input data and intermediate data stored in the data buffer 318 are reused across multiple operations in the neural engines 314, thereby reducing data transfers to and from the system memory 230. The data buffer 318 may be operated in a broadcast mode, where input data for all input channels is provided to all neural engines 314, or in a unicast mode, where input data for a subset of the input channels is provided to each neural engine 314.

[0040] The input data 322 stored in the data buffer 318 may be, among other things, image data from a previous cycle of the neural engine 314, histogram of brightness gradient (HOG) data, audio data, metadata, output data 328, and portions of other processed data received from other components of the SOC component 204.

[0041] The buffer DMA 320 includes a read circuit that receives a portion (e.g., a tile) of input data from a source (e.g., system memory 230) for storage in the data buffer 318, and a write circuit that delivers the data from the data buffer 318 to a target (e.g., system memory). Exemplary Neural Engine Architecture

[0042] 4 is a block diagram of a neural engine 314 according to one embodiment. The neural engine 314 performs various operations to facilitate neural network operations, such as convolution, spatial pooling, and local response normalization. The neural engine 314 receives input data 322, performs multiply-accumulate operations (e.g., convolution operations) on the input data 322 based on stored kernel data, performs further post-processing operations on the results of the multiply-accumulate operations, and generates output data 328. The input data 322 and / or output data 328 of the neural engine 314 may be single channel or multi-channel.

[0043] The neural engine 314 may include, among other components, an input buffer circuit 402, a computation core 416, a neural engine (NE) control 418, a kernel extraction circuit 432, an accumulator 414, and an output circuit 424. The neural engine 314 may include additional components not shown in FIG.

[0044] The input buffer circuit 402 is a circuit that receives input data 322 from the data buffer 318 while storing a portion of the input data and sending a portion of the input data 408 appropriate for a current task or process loop to the computation core 416 for processing. The input buffer circuit 402 includes a shifter 410 that shifts the read location of the input buffer circuit 402 to change the portion of the input data 408 sent to the computation core 416. By changing the portion of the input data provided to the computation core 416 via shifting, the neural engine 314 can perform multiply-accumulate on different portions of the input data based on a fewer number of read operations. In one or more embodiments, the input data 322 includes data for difference convolution groups and / or input channels.

[0045] The kernel extraction circuit 432 is a circuit that receives the kernel data 326 from the kernel DMA 324 and extracts the kernel coefficients 422. In one embodiment, the kernel extraction circuit 432 references a look up table (LUT) and uses a mask to reconstruct the kernel from the compressed kernel data 326. The mask indicates locations in the reconstructed kernel that are to be filled with zeros, and the remaining locations are filled with numbers. The kernel coefficients 422 of the reconstructed kernel are sent to the computation core 416 and loaded into registers in a multiply-add (MAD) circuit of the computation core 416. In another embodiment, the kernel extraction circuit 432 receives the kernel data in an uncompressed format and the kernel coefficients are determined without reference to a LUT or use of a mask.

[0046] Computation core 416 is a programmable circuit that performs computational operations. To this end, computation core 416 may include MAD circuits MAD0-MADN and a post-processor 428. Each of MAD circuits MAD0-MADN may store an input value in input data portion 408 and a corresponding kernel coefficient in kernel coefficients 422. The input value and the corresponding kernel coefficient are multiplied in each MAD circuit to generate processed value 412.

[0047] The accumulator 414 is a memory circuit that receives and stores the processed values ​​412 from the MAD circuit. The processed values ​​stored in the accumulator 414 may be sent as feedback information 419 for further multiplication and addition operations in the MAD circuit, or may be sent to a post-processor 428 for post-processing. The accumulator 414 in combination with the MAD circuit forms a multiply-accumulator (MAC) 404. In one or more embodiments, the accumulator 414 may have sub-units, each of which sends data to a different component of the neural engine 314. For example, during a processing cycle, data stored in a first sub-unit of the accumulator 414 is sent to the MAC circuit, and data stored in a second sub-unit of the accumulator 414 is sent to the post-processor 428.

[0048] The post-processor 428 is a circuit that performs further processing of the values ​​412 received from the accumulator 414. The post-processor 428 may perform operations including, but not limited to, non-linear functions (e.g., Rectified Linear Unit (ReLU)), normalized cross-correlation (NCC), merging the results of performing neural operations on 8-bit data with 16-bit data, and local response normalization (LRN). The results of such operations are output from the post-processor 428 to the output circuit 424 as processed values ​​417.

[0049] The NE control unit 418 controls the operation of the other components of the neural engine 314 based on the operating mode and parameters of the neural processor circuit 218. Depending on different operating modes (e.g., group convolution mode or non-group convolution mode) or parameters (e.g., number of input channels and number of output channels), the neural engine 314 may operate in different sequences on different input data, return different values ​​from the accumulator 414 to the MAC circuit, and perform different types of post-processing operations in the post-processor 428. To configure the components of the neural engine 314 to operate in a desired manner, the NE control unit 418 sends control signals to the components of the neural engine. The NE control unit 418 may also include a rasterizer 430 that keeps track of the current task or process loop being processed by the neural engine 314, as will be described in more detail below with reference to Figures 5-7.

[0050] The output circuitry 424 receives the processed values ​​417 from the post processor 428 and interfaces with the data buffer 318 to store the processed values ​​417 in the data buffer 318. To this end, the output circuitry 424 may deliver as output data 328 in sequence, out of sequence, or in a form in which the processed values ​​417 are processed in the post processor 428.

[0051] Components within the neural engine 314 may be configured by the NE controller 418 and the neural task manager 310 during configuration. To this end, the neural task manager 310 sends configuration information to the neural engine 314 during configuration. Configurable parameters and modes include, but are not limited to, mapping between input data elements and kernel elements, number of input channels, number of output channels, execution of output stride, and enabling / selecting post-processing operations in the post processor 428. Operation of segmenting data for processing in a neural processor circuit

[0052] Input data is typically divided into smaller pieces of data for parallel processing on multiple neural engines 314. Often, multiple cycles of operations are performed to generate an output for a task associated with a neural network. A compiler executed by CPU 208 analyzes the hierarchy and nodes of the neural network and determines how the input data is segmented based on the hardware constraints of the neural processor circuit 218. One of the compiler's functions is to determine how the input data should be divided into smaller data units for processing in the neural engines 314 and how the processing should be repeated in a loop to generate a result for the task.

[0053] 5 is a conceptual diagram illustrating loops for processing input data in the neural processor circuit 218, according to one embodiment. The outermost loop represents processing on a convolution group when group convolution with multiple convolution groups is used. A group convolution is a convolution in which the input data of the input channels in each group is used only to generate output data for the output channels of each group, but is not used to generate output data for the output channels of other groups. Thus, each group of a group convolution can be treated as a separate convolution operation.

[0054] In each convolution group loop, there is a processing loop for a slice of input data. The entire input data for the convolution operation is segmented into multiple strips of slices in an overlapping fashion as shown in FIG. 6. The overlapping portions 602, 604, 606 are portions of the input data that are over-fetched in two adjacent slices to provide spatial support for the corresponding kernel. The second outermost loop performs the convolution operation for each slice in the input data. In the loop for a slice, there is a processing loop for tiles of the slice. Each slice is segmented into multiple tiles as shown in FIG. 6. The overlapping portions 608, 610, 612, 614 are portions of the input data in slice 4 that are over-fetched in two adjacent tiles to provide spatial support for the corresponding kernel. The right-most tile will typically have a smaller width than the other tiles of the slice. In one embodiment, the input data for each tile is loaded onto the data buffer 318 in a read cycle and reused for operations in the processing loop for the tile. Within the processing loop for tiles is a processing loop for work units. Each tile is segmented into multiple work units as shown in FIG. 6. A work unit is a portion of input data that has a size that produces an output value that fits into the accumulator 414 of the neural engine 314 during a single cycle of the computation core 416. Although the shape of each work unit is shown as a horizontal strip in FIG. 6, the shape of the work unit may vary depending on the shape and size of the tile. The work units also have overlapping portions that represent over-fetched data to provide support for the corresponding kernel. In particular, the work unit of the last tile of a slice may have the shape of a vertical strip if the tile is vertical. In one or more embodiments, the size of each work unit is 256 bytes. In such embodiments, for example, the work units are shaped into one of the following dimensions: 16×16, 32×8, 64×4, 128×2, or 256×1.

[0055] For each work unit, an internal processing loop for an output channel group (OCG) may be provided. The number of output channels generated for a given work unit by a single cycle of the computational core 416 is referred to as an OCG. Depending on the mode of operation, each neural engine 314 may process output data for a different number of output channels (e.g., 8 channels, 32 channels) for a single load of input data into the input buffer circuit 402.

[0056] For each output channel group, an inner processing loop for the input channel (Cin) may be provided. If an input stride is implemented to skip certain input data, a loop for a sub-input channel (Sub-Cin) may be provided within the processing loop for the input channel (Cin).

[0057] For each input channel or sub-input channel, an inner loop is provided to process the horizontal spatial support for the kernel and the vertical support within each horizontal spatial support. The spatial support refers to the input data for convolution with the kernel and includes over-fetched input data at the end of the input data for performing the convolution.

[0058] Over-fetching refers to fetching additional input data within a current slice, tile, or work unit so that appropriate dimensions of the input data can be provided for convolution with a kernel. In one or more embodiments, over-fetching is performed vertically between slices to obtain additional rows of input data (shown in FIG. 6 as overlapping portions 602, 604, 606), horizontally between tiles to obtain additional columns of input data (shown in FIG. 6 as overlapping portions 608, 610, 612, 614), and vertically between work units within a tile to obtain additional rows of input data.

[0059] For each spatial support for the kernel, an inner processing loop for an output channel (OC) is provided to generate output data for each output channel (Cout). If the output stride implements spatial upsampling, an additional inner loop is provided to process each sub-output channel. If the output stride is implemented, the loading of kernel coefficients and MAC operations are performed in the loop for the output channel (OC) or sub-output channel to generate output data for the output channel (OC) or sub-output channel.

[0060] The nested loop structure of Figure 5 is merely illustrative. Loops may be omitted, added, or restructured depending on various factors. For example, if only a single convolution group is used, the outermost loop may be removed. Additionally, the loop structures for horizontal and vertical spatial support may be reversed.

[0061] In one or more embodiments, the operations associated with dividing the input space into smaller units and processing these smaller units, as described above with reference to Figures 5 and 6, are performed by rasterizers 714, 718, 720, 722 in the various components of the neural processor circuit 218. The rasterizers are circuits in the various components of the neural processor circuit 218 that keep track of segments of input / output data (e.g., groups, work units, input channels, output channels) and instruct the components of the neural processor circuit for appropriate handling of the segments of input data. For example, the rasterizer 720 in the buffer DMA 320 keeps track of tiles and slices received from the system memory 230, while the rasterizer 718 in the data buffer 318 broadcasts work units in order for processing by the neural engine 314. A rasterizer 722 in the kernel DMA 322 determines which kernels are received and distributed to the neural engine 314, while a rasterizer 714 in the neural engine 314 operates the shifter 410 in the input buffer circuit 402 to deliver the correct portions of the input data 408 to the MAC 404 and send the final output data 328 to the data buffer 318.

[0062] 7 illustrates the programming of rasterizers 714, 718, 720, 722 within components 314, 318, 320, 322 of neural processor circuit 218, according to one embodiment. To perform their functions, each of rasterizers 714, 718, 720, 722 receives task information 710 (e.g., configuration data) that indicates how input data and / or kernel data should be segmented and handled by each component of neural processor circuit 218. The task information includes information about the details of the current layer (e.g., dimensions of input and output data, dimensions of associated kernels, type of padding at the borders of input data). Rasterizers 714, 718, 720, and 722 may also receive constraints on their operation (e.g., whether to allow or not allow tile widths above a threshold).

[0063] Providing rasterizers in different components of the neural processor circuit 218 can reduce overhead in data communicated between components of the neural processor circuit 218. If a single central rasterizer is provided to control different components of the neural processor circuit 218, this data may require kernel data, input data, and output data communicated between components to identify associated positions in a loop of a task, such as convolution groups, tiles, slices, work units, input channels, and output channels. By using distributed rasterizers, separate metadata is not required to communicate kernel data, input data, and output data between components of the neural processor circuit 218. Example Process in Neural Engine Architecture

[0064] 8 is a flow chart illustrating one embodiment of a method for processing input data in the neural processor circuit 218. The method may include different and / or additional steps, or the steps may be in a different order.

[0065] After neural task manager 310 has programmed rasterizers 714, 718, 720, 722, the process of operating buffer DMA 320 begins by rasterizer 720 instructing buffer DMA 320 to receive (804) tiles of input data from system memory 230. The tiles received by buffer DMA 320 are stored (806) in data buffer 318.

[0066] The rasterizer 718 in the data buffer 318 then instructs the data buffer 318 to send the work unit to one or more neural engines 314 (808). The work unit is then stored in the input buffer circuit 402 of the one or more neural engines 314.

[0067] In one or more embodiments, the input buffer circuit 402 selects (816) a portion of the work unit to be sent to the MAC 404 to perform a multiply-accumulate operation. The MAC 404 then performs (820) the multiply-accumulate operation on the selected portion of the work unit using a corresponding kernel. It then determines (824) whether the entire work unit has been processed in one or more neural engines 314. If the determination is no, the selected portion of the work unit is shifted (828) by the shifter 410 and the multiply-accumulate operation is performed again (820).

[0068] If it is determined (824) that the entire work unit has been processed, the process proceeds to determine whether all work units in the tile have been processed (832). If the determination is no, the process proceeds to the next work unit (836) by causing the data buffer 318 to send the next work unit to one or more neural engines 314 (808) and repeats the subsequent process.

[0069] If it is determined (832) that all work units in the tile have been processed by the neural engine 314, the process proceeds to determine whether all tiles of input data have been processed (840). If the determination is no, the process proceeds to the next tile (844) by having the rasterizer 720 instruct the buffer DMA 320 to receive the next tile from the system memory 230 (804), and the subsequent process repeats.

[0070] If it is determined that all tiles of the input data have been processed (840), the process ends for the current input data. The process may then be repeated to process the next input data, or may proceed to the next task.

[0071] The process embodiment as described above with reference to Figure 8 is merely exemplary. Additional loops may be implemented as described above with reference to Figure 5. Moreover, the sequence of processes may be modified or omitted. Exemplary Neural Task Manager Architecture

[0072] A neural network may include network layers or sub-layers that are instantiated or implemented as a series of tasks executed by the neural processor circuit 218. Figure 9 is a schematic block diagram illustrating a neural network 900 represented by a list of tasks 904 according to one embodiment. The neural network 900 includes convolutional layers C1, C2, C3 (sub-layer C3 00 , C3 10 , C3 11 , C3 20 , and C3 219, C1, C2, and C3, as well as network layers (or sub-layers) including C, C4, C5, and C6, and pooling layers P2 and P4. Neural network 900 is one example of a neural network architecture that may be instantiated by neural processor circuit 218, i.e., when neural network 900 is converted into a task list 904 and executable by neural processor circuit 218. Other types of neural network architectures having different types of network layers or different orders of network layers may also be instantiated by neural processor circuit 218.

[0073] The neural network 900 is converted into a task list 904, for example by the CPU 208. The task list 904 includes the following tasks: task C1, task C2+P2, task C3 00 +P4, Task C3 10 , Task C3 11 +P4, Task C3 20 , Task C3 21 +P4, Task C5 a , Task C5 b , and task C6. Each task is associated with a task descriptor that defines a configuration of the neural processor circuit 218 for performing the task. Each task may correspond to a single network layer of the neural network 900, a portion of a network layer of the neural network 900, or multiple network layers of the neural network 900. For example, task C1 corresponds to a single network layer C1, task C2+P2 corresponds to multiple network layers C2 and P2, task C5 corresponds to a single network layer C6, task C6 corresponds to a network layer C7, task C8 corresponds to a network layer C9, task C1 corresponds to a network layer C1, task C2+P2 corresponds to multiple network layers C2 and P2, task C5 corresponds to a network layer C1, task C6 corresponds to a network layer C2, task C7 corresponds to a network layer C4, task C8 corresponds to a network layer C5, task C9 corresponds to a network layer C6, task C1 corresponds to a network layer C1, task C1 corresponds to a network layer C2, task C1 corresponds to a network layer C1, task C1 corresponds to a network layer C2, task C1 corresponds to a network layer C3, task C1 corresponds to a network layer C4, task C2+P2 corresponds to multiple network layers C2 and P2, task C5 corresponds to a network layer C5, task C6 corresponds to a network layer C7, task C8 corresponds to a network layer C8, task C1 corresponds to a network layer C1, task C1 corresponds to a network layer C1, task C1 corresponds to a network layer C2, task C1 corresponds to a network layer C1, task C1 corresponds to a network layer C2, task C1 corresponds to a network layer C3, task C1 corresponds to a network layer C4, task C1 corresponds to a network layer C5, task C2 corresponds to a network layer C6, task C1 corresponds to a network layer C1, task C2+P2 corresponds to multiple network layers C2 and P2, task C5 corresponds to a network layer C1, task C1 corresponds to a network layer C1, task C1 corresponds to a network layer C2, task C1 corresponds to a network a and C5 b each corresponds to a portion of network layer C5. The neural processor circuit 218 instantiates the neural network 900 by executing tasks in a task list 904 under the control of the neural task manager 310.

[0074] 10 is a block diagram illustrating a neural task manager 310, according to one embodiment. The neural task manager 310 manages the execution of tasks for one or more neural networks 900 by the neural processor circuit 218. The neural task manager 310 may include, among other components, a task arbiter 1002, task queues 1004A-1004N (collectively referred to herein as "task queues 1004" and individually as "task queues 1004"), a task manager direct memory access (DMA) 1006, a fetch queue 1008, and a configuration queue 1010. The neural task manager 310 may include other components not shown in FIG.

[0075] The task arbiter 1002 is a circuit, or a combination of circuitry and firmware, executed by the neural processor circuit 218 that selects a task from the task queue 1004. The task arbiter 1002 dequeues a task from the task queue 1004 and places the task in the configuration queue 1010. While the task is in the configuration queue, the task is committed to execution and the neural processor circuit performs prefetching on input and kernel data before the task is executed by other components of the neural processor circuit 218. For example, the task arbiter 1002 may perform fixed priority arbitration between multiple task queues 1004 and select the task with the highest priority from the task queue 1004 for acquisition of a task descriptor 1012 from the system memory 230 by the task manager DMA 1006.

[0076] The neural task manager 310 may include one or more task queues 1004. Each task queue 1004 is coupled to the CPU 208 and the task arbiter 1002. Each task queue 1004 receives a reference from the CPU 208 to a task list 904 of a task that, when executed by the neural processor circuit 218, instantiates the neural network 900. The reference stored in each task queue 1004 may include a set of pointers and counters that point to the task list 904 in a task descriptor 1012 in the system memory 230. Each task queue 1004 may further be associated with a priority parameter that defines the relative priority of the task queue 1004. The task descriptor 1012 of a task specifies the configuration of the neural processor circuit 218 for executing the task.

[0077] Task manager DMA 1006 is coupled to task arbiter 1002, system memory 230, and fetch queue 1008. Task manager DMA 1006 includes read circuitry that receives task descriptors 1012 for tasks from a source (e.g., system memory 230) for storage in fetch queue 1008. For example, task arbiter 1002 selects a task queue 1004 according to the priority of the task queue 1004, and controls task manager DMA 1006 to select task descriptors 1012 for the tasks using a task list 904 referenced by the selected task queue 1004.

[0078] The fetch queue 1008 is a single entry queue that stores task descriptors 1012 for tasks that are pending commitment to execute. The fetch queue 1008 is coupled to the task manager DMA 1006 to receive task descriptors 1012 from the system memory 230 and provide the task descriptors 1012 to the configuration queue 1010 or provide configuration data 1014 extracted from the task descriptors 1012 to the configuration queue 1010.

[0079] The configuration queue 1010 holds configuration data 1014 for multiple tasks that are committed to execution. If a task is in the configuration queue 1010, the kernel DMA 324 may fetch kernel data from the system memory 230 to store in the kernel extraction circuit 432 of the neural engine 314, and the buffer DMA 320 may fetch input data from the system memory 230 to store in the data buffer 318. To execute the task, the kernel extraction circuit 432 provides prefetched kernel data to the MAC 404 of the neural engine 314, and the data buffer 318 provides prefetched input data to the MAC 404 of the neural engine 314. In some embodiments, the configuration queue 1010 may include multiple queues that hold configuration data 1014 extracted from the committed task descriptors 1012. As discussed in more detail in connection with FIG. 13, the configuration queue 1010 is further coupled to other components of the neural processor circuit 218 to configure the neural processor circuit 218 in response to the configuration data 1014.

[0080] 11 is a diagram illustrating retrieval of a task descriptor 1012 using a task queue 1004, according to one embodiment. The task queue 1004 includes references, e.g., a set of pointers, to task descriptors 1012A-1012N stored in system memory 230. To that end, the task queue 1004 may include memory that stores a head parameter 1102, a network identifier (ID) 1104, a base address index 1106, a tail parameter 1108, a count parameter 1110, and a priority parameter 1112. The head parameter 1102 is a pointer to a location in system memory 230 that stores task descriptor 1012A, which is at the head of the task queue 1004. The network ID 1104 identifies the neural network 900 of the task descriptor 1012 at the head of the task queue 1004, and the base address index 1106 is an index into a base address table 1114 tagged with the network ID 1104 of the task descriptor 1012A at the head of the task queue 1004. The count parameter 1110 defines the number of task descriptors 1012 in the task queue 1004. The priority parameter 1112 defines the priority of the task queue 1004 and is used by the task arbiter 1002 to select among multiple task queues 1004.

[0081] If a particular task queue 1004 is selected (e.g., in response to the priority parameter 1112), the task arbiter 1002 references the head parameter 1102, the network ID 1104, the base address index 1106, and the base address table 1114 to obtain a task descriptor 1012 from the system memory 230 and place the task descriptor 1012 in the fetch queue 1008 to initiate a commitment for task execution. At each configuration period, the task arbiter 1002 may continue to place task descriptors 1012 in the fetch queue 1008 in response to the order of tasks defined by the task list 904 of the task queue 1004, such as by obtaining the next task descriptor 1012B.

[0082] FIG. 12 illustrates a task descriptor 1012, according to one embodiment. The task arbiter 1002 places the task descriptor 1012 from the system memory 230 in a fetch queue 1008, which is then forwarded to a configuration queue 1010. The task descriptor 1012 with the highest priority (e.g., entered first) in the configuration queue 1010 is used to configure the neural processor circuit 218 for execution during a configuration period. The task descriptor 1012 includes a task descriptor header 1202 and configuration data 1014 including address data 1204A-1204N (hereinafter referred to as "address data 1204"). The task descriptor header 1202 includes configuration data 1014 that configures various operations of the neural task manager 310, including operations related to task selection and task switching. For example, the task descriptor header 1202 may be parsed by the task arbiter 1002 to extract configuration data 1014 that programs the neural task manager 310 and other components of the neural processing circuit 218. The task descriptor header 1202 may include a task identifier (ID) 1206 that identifies the task, a neural network identifier (ID) 1208 that identifies the neural network 900 instantiated by the task, task switch parameters 1210 that define whether the neural task manager 310 should initiate a task switch after execution of the task (e.g., to execute a task in a different task queue 1004), input surface parameters 1212 that define whether input data for the task should be obtained from the system memory 230 or the data buffer 318, output surface parameters 1214 that define whether output data for the task should be stored in the system memory 230 or the data buffer 318, various (e.g., base address) pointers 1216 to facilitate programming of the neural processor circuit 218, and one or more debug / exception parameters 1218 that control event, exception, or debug logging.

[0083] Each instance of address data 1204A-1204N (collectively or individually referred to as "address data 1204") defines an address and data payload pair used to program components of the neural processor circuit 218. The data payload may include input data and kernel data used to execute a task. For example, each instance of address data 1204 includes register data defining the data payload, a register address defining a destination memory location in the neural processing circuit 218 to receive the register data, and a register count defining consecutive memory locations (e.g., registers) to be written with the register data. In some embodiments, the register address is combined with a base address stored in the header 1202 to define the full address of each memory location. If the task descriptors 1116 are generated at compile time, the actual runtime addresses may not be known. The base address table 1114 is used to avoid all task descriptors being duplicated or updated with dynamically allocated addresses.

[0084] 13 is a block diagram illustrating a fetch queue 1008 and a configuration queue 1010, according to one embodiment. The configuration queue 1010 is coupled to the fetch queue 1008, which is coupled to the system memory 230 via the task manager DMA 1006. The configuration queue 1010 is further coupled to one or more of the rasterizers 714 of the neural engine 314, the rasterizers 718 of the data buffer 318, the rasterizers 720 of the buffer DMA 320, and the rasterizers 722 of the kernel DMA 322. The fetch queue 1008 stores task descriptors 1012 (e.g., including task descriptor headers 1202 and address data 1204A-1204N) for tasks that are pending and not committed to execution. The fetch queue 1008 reduces the latency of reading the next task descriptor 1012 from the system memory 230 into the configuration queue 1010. The fetch queue 1008 stores the task descriptor 1012 with the highest priority as determined by the task arbiter 1002. The task arbiter 1002 may replace a task descriptor 1012 stored in the fetch queue 1008 if a higher priority task descriptor 1012 is enqueued (e.g., from a higher priority task queue 1004). A task descriptor 1012 in the fetch queue 1008 does not initiate prefetching of input data or kernels and does not affect task queue priorities, pointers, or counters. Thus, a task descriptor 1012 in the fetch queue 1008 may be easily replaced by a higher priority task descriptor 1012 by writing the higher priority task descriptor 1012 to the fetch queue 1008. When a task descriptor 1012 stored in the configuration queue 1010 has been executed by the neural processor circuit 218, the task descriptor 1012 stored in the fetch queue 1008 may be transferred to the configuration queue 1010 and another task descriptor 1012 for a subsequent task may be stored in the fetch queue 1008.

[0085] The configuration queue 1010 stores task descriptors 1012 for tasks that have been committed for execution by the neural processor circuit 218. In some embodiments, the configuration queue 1010 includes multiple separate queues 1310, each of which stores a portion of the configuration data 1014 (including configuration data 1014A-1014E) extracted from the task descriptors 1012. Additionally, each of the queues 1310 is coupled to a corresponding component of the neural processor circuit 218 to program that component with the configuration data 1014. Through operation of the configuration queue 1010, the neural task manager 310 programs the rasterizers 714, 718, 720, 722 to perform the functions described above in Figures 7 and 8. For example, queue 1310A is coupled to rasterizer 714 of neural engine 314 to provide configuration data 1014A that controls operation of shifter 410 in input buffer circuit 402 to deliver correct portions of input data 408 to MAC 404 and send final output data 328 to data buffer 318. Queue 1310B is coupled to rasterizer 718 of data buffer 318 to provide configuration data 1014B that controls broadcast of input data (e.g., work units) by data buffer 318 for processing by neural engine 314. Queue 1310C is a read queue coupled to rasterizer 720 of buffer DMA 320 to provide configuration data 1014C that controls buffer DMA 320 to retrieve input data (e.g., tiles) from system memory 230 and store the input data in data buffer 318. Queue 1310D is a write queue coupled to rasterizer 720 of buffer DMA 320 and provides configuration data 1014D that controls buffer DMA 320 to store output data in system memory 230. Queue 1310E is coupled to rasterizer 722 of kernel DMA 322 and provides configuration data 1014E that controls which kernels are received and distributed to neural engine 314. In some embodiments, task descriptors 1012 or configuration data 1014 stored in configuration queue 1010 cannot be replaced and will be executed in first-in, first-out (FIFO) order. Example Processes in the Neural Task Manager Architecture

[0086] 14 is a flow chart illustrating one embodiment of a method for managing tasks in the neural processor circuit 218. The method may include different and / or additional steps, or the steps may be in a different order.

[0087] The CPU 208 generates (1402) a task list 904 of task descriptors 1012 of tasks that, when executed by the neural processor circuit 218, instantiate the neural network 900. For example, the CPU 208 may receive input data for the machine learning operation from the image sensor 202, the system memory 230, the persistent storage 228, the network interface 210, or some other component. The machine learning operation may include an inference operation or a training operation. The neural network 900 may include a neural network architecture including kernel data and network layers. The input data is applied to the neural network 900 to perform the machine learning operation. The kernel data and network layers of the neural network 900 may be calculated in a machine learning training process. The CPU 208 performs a compilation operation (offline or on the fly) to turn the neural network description 900 into a linked list of task descriptors 1012, referred to herein as a task list 904. Each task is defined by a task descriptor 1012 and, when executed by the neural processor circuit 218, instantiates a single network layer, multiple network layers, or a portion of a network layer. Each task descriptor 1012 for a task includes configuration data 1014, such as a task descriptor header 1202, and address data 1204 that defines a neural processor circuit 218 address and data payload pair. The data payload may include kernel data and input data for the neural network 900. The configuration data 1014 further includes instructions that configure the operation of the rasterizers 714, 718, 720, and 722 to perform the task.

[0088] The CPU 208 stores 1404 the task list 904 of task descriptors 1012 in the system memory 230. In some embodiments, the CPU 208, or another CPU external to the electronic device 100, generates the task descriptors 1012 and stores the task descriptors 1012 in the persistent storage 228 or some other non-volatile memory. The task descriptors 1012 are loaded along with kernel and input data in the system memory 230 for use by the neural processor circuit 218. The CPU 208 may be coupled to the system memory 230 via a bus 232 and a memory controller 222.

[0089] The task list 904 may include a set of pointers that reference the location of the task descriptors 1012 in the system memory 230. Additionally, the CPU 208 may update parameters of the task descriptors 1102, such as memory addresses or network IDs. For example, the task list 904 may include head parameters 1102 and tail parameters 1108 that define the beginning and ending register addresses, respectively, of the system memory 230 where the task descriptors 1012 are stored. In some embodiments, the references to register addresses in the task list 904 may be partial addresses, and a base address table 1114 is used to define the full reference address into the system memory 230. In some embodiments, the CPU 208 may patch absolute addresses as needed. The CPU 208 may further configure the neural processor circuit 218 by setting its base address registers.

[0090] A task arbiter 1002 of the neural task manager 310 enqueues 1406 task lists 904 for execution by the neural processor circuit 218. For example, the neural task manager 310 includes a number of task queues 1004A-1004N. Each of the task queues 1004 may store a reference to a task list 904. Further, the task queues 1004 are prioritized for execution according to a priority parameter 1112 of the task list 904 referenced by the task queue 1004.

[0091] In some embodiments, the CPU 208 executes a general configuration of the neural processor circuit 218 to perform a task. The CPU 208 may also start the neural processor circuit 218 if it is not already running.

[0092] The task arbiter 1002 selects 1408 the task queue with the highest priority for execution. For example, the task arbiter 1002 selects a task queue 1004 during each programming period based on a comparison of the priority parameters of the task queues 1004 or a comparison of the task queues 1004 with the task lists 904, and executes a task from the task list 904 starting with the task queue 1004 with the highest priority.

[0093] The neural processor circuit 218 executes (1410) the task from the selected task queue. For example, the neural processor circuit 218 executes the method of FIG. 8 to execute the task. The configuration queue 1010 of the neural task manager 310 may provide configuration data 1014 (or task information 710) to the rasterizers 714, 718, 720, 722 of the neural processor circuit 218 to program the neural processor circuit 218 to execute the task, as shown in FIG. 7. Furthermore, the execution of the task may include processing prefetched kernel data and input data while the task is in the configuration queue 1010. The execution of each task may include multiple processing loops to handle the input data, as shown in FIGS. 5 and 6. The task arbiter 1002 may dequeue the task from the configuration queue 1010 after the task is executed.

[0094] In some embodiments, to execute a task, the task arbiter 1002 controls the task manager DMA 1006 to obtain a task descriptor 1012 for the task in the task list 904 and store the task descriptor 1012 in the fetch queue 1008. After execution of a previously committed task, the task arbiter 1002 may dequeue the executed task by removing the task's task descriptor or configuration data from the configuration queue 1010. The task descriptor 1012 or extracted configuration data 1014 for the current task is then placed from the fetch queue 1008 to the configuration queue 1010. When the task is in the configuration queue 1010, the neural processor circuit 218 may initiate a prefetch operation by the kernel DMA 324 for kernel data from the system memory 230 to the kernel extraction circuit 432 and a prefetch operation by the buffer DMA 320 for input data from the system memory 230 to the data buffer 318.

[0095] If a different task queue has a higher priority, a task switch to the task queue with the higher priority may be performed if possible. To perform a task switch (e.g., to another machine learning operation), the task arbiter 1002 replaces a task in the fetch queue 1008 with another task descriptor referenced in the different task queue 1004. The task arbiter 1002 may resume the interrupted machine learning operation defined by the task list 904 after the task switch is completed by storing the replaced task in the fetch queue 1008.

[0096] In some embodiments, the CPU 208 determines 1408 a priority parameter 1110 of the task list 904 to select a task queue. The priority parameter 1110 defines the priority of the task of the machine learning operation relative to other tasks of other machine learning operations executed by the neural processor circuit 218. The CPU 208 may facilitate the execution of multiple machine learning operations. The machine learning operations may differ, such as by using different input data or different neural network architectures and kernel data. The CPU 208 may determine the priority parameter of the task list 904 programmatically or based on user input. The task descriptor 1012 referenced by the task queue 1004 with the highest priority may be selected for execution. The priority of the task queue may be determined by the CPU 208 or dynamically determined by information from previously executed task descriptors.

[0097] The task arbiter 1002 determines (1412) whether all tasks in the task list 904 have been executed. If no tasks in the task list 904 have been executed, the process returns to step 1410, where the task arbiter 1002 executes any tasks in the task list 904 that have not been executed.

[0098] As each task in the task list 904 is executed, the task arbiter 1002 removes 1414 the task list 904 from the task queue 1004. In another embodiment, the reference to the task's task descriptor 1012 may be removed from the task queue 1004 after the task is executed. The process may end. In another embodiment, the process may return to 1402, where the CPU 208 may continue to generate task descriptors for tasks for other machine learning operations. In that sense, the process may be repeated to place tasks in the task queue according to the specified task priority for execution of the machine learning operations by the neural processor circuit 218.

[0099] While particular embodiments and applications have been illustrated and described, it should be understood that the invention is not limited to the precise construction and components disclosed herein, and various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope of the disclosure.

Claims

1. A neural engine circuit, a neural task manager circuit coupled to the neural engine, the neural task manager comprising: a task queue circuit configured to store a reference to a task list containing tasks for instantiating a neural network, the task list including configuration data for tasks stored in a memory location external to the neural processor circuit; a task arbiter circuit coupled to the task queue circuit, the task arbiter circuit comprising: retrieving the configuration data for the task from the location in the memory external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; configured to provide a portion of the configuration data to the neural engine circuit; a task arbiter circuit, wherein the portion of the configuration data programs the neural engine circuit to perform the task. Neural processor circuit.

2. 2. The neural processor circuit of claim 1, wherein the tasks, when executed, instantiate a single network layer of the neural network, multiple network layers of the neural network, or a portion of a network layer of the neural network.

3. the task arbiter circuit is further configured to store the configuration data in a configuration queue of the neural task manager circuit, the configuration queue being coupled to the neural engine circuit and configured to provide the portion of the configuration data to the neural engine circuit; The neural processor circuit comprises: a kernel direct memory access (DMA) configured to retrieve kernel data for the task from the memory external to the neural processor circuit when the configuration data is stored in the configuration queue; a buffered direct memory access (DMA) configured to obtain input data for the task from the memory external to the neural processor circuit when the configuration data is stored in the configuration queue; 2. The neural processor circuit of claim 1 further comprising:

4. The neural task manager circuit includes: a fetch queue coupled to the configuration queue; a task manager direct memory access (DMA) coupled to the fetch queue and to the task arbiter circuit; Further comprising: the task arbiter circuit is further configured to retrieve, via the task manager DMA, the configuration data for the task from the location of the memory external to the neural processor circuit, and store the configuration data in the fetch queue, the fetch queue providing the configuration data to the configuration queue when a second configuration data for an executed task is removed from the configuration queue.

4. The neural processor circuit of claim 3.

5. the neural task manager circuit further includes another task queue circuit coupled to the task arbiter circuit, the other task queue circuit configured to store another reference to another task list for another task instantiating another neural network, the task queue circuit having a first priority and the other task queue circuit having a second priority; the task arbiter circuitry is configured to store one of: (i) the configuration data for the task; or (ii) other configuration data for the other task in a fetch queue based on a comparison of the first priority and the second priority; 4. The neural processor circuit of claim 3.

6. 2. The neural processor circuit of claim 1, further comprising a data buffer coupled to the memory and to the neural engine circuit that is external to the neural processor circuit, the task arbiter circuit being further configured to provide another portion of the configuration data to the data buffer, the other portion of the configuration data programming the data buffer to broadcast work units of input data for the task to the neural engine circuit.

7. a buffer direct memory access (DMA) coupled to the data buffer and to the memory external to the neural processor circuit; the task arbiter circuit is further configured to provide a third portion of the configuration data to the buffer DMA, the third portion of the configuration data programming the buffer DMA to obtain tiles of the input data for the task from the memory external to the neural processor circuit and to store the tiles in the data buffer; The neural processor circuit of claim 6 , wherein the tile comprises a plurality of work units.

8. The neural engine circuit comprises: an input buffer circuit coupled to the neural task manager; a multiply-add (MAD) circuit coupled to the input buffer circuit; Including, the portion of the configuration data programs the input buffer circuit to provide the MAD circuit with the portion of the input data for the task stored in the input buffer circuit; 2. The neural processor circuit of claim 1.

9. 9. The neural processor circuit of claim 8, further comprising a data buffer coupled to said memory and to said neural engine circuit external to said neural processor circuit, said neural engine further comprising an output circuit, and said at least a portion of said configuration data programs said output circuit to provide output data from said MAD circuit to said data buffer.

10. a kernel direct memory access (DMA) coupled to the memory external to the neural processor circuit and to the neural engine circuit; 2. The neural processor circuit of claim 1, wherein the task arbiter circuit is further configured to provide another portion of the configuration data to the kernel DMA, the other portion of the configuration data programming the kernel DMA to retrieve kernel data from the memory external to the neural processor circuit and provide the kernel data to the neural engine circuit to perform the task.

11. 1. A method of managing tasks in a neural processor circuit, comprising: storing, in a task queue circuit of the neural processor circuit, a reference to a task list of tasks for instantiating neural networks, the task list including configuration data for tasks stored in a memory location external to the neural processor circuit; retrieving the configuration data from the location of the memory external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; providing a portion of the configuration data to a neural engine circuit of the neural processor circuit, the portion of the configuration data programming the neural engine circuit to perform the task; The method includes:

12. The method of claim 11 , wherein the task, when executed, instantiates a single network layer of the neural network, multiple network layers of the neural network, or a portion of a network layer of the neural network.

13. storing said configuration data in a configuration queue of said neural processor circuit; providing said portion of said configuration data from said configuration queue to said neural engine circuit; retrieving kernel data for the task from an external memory if the configuration data is stored in the configuration queue; obtaining input data for the task from the external memory if the configuration data is stored in the configuration queue; The method of claim 11 further comprising:

14. Retrieving the configuration data for the task from the location in the external memory via a task manager direct memory access (DMA) of the neural processor circuit; storing said configuration data in a fetch queue of said neural processor circuit; providing configuration data from the fetch queue to the configuration queue when other configuration data for an executed task is removed from the configuration queue; The method of claim 13 further comprising:

15. storing another reference to another task list of another task instantiating another neural network in another task queue circuit, the task queue circuit having a first priority and the other task queue circuit having a second priority; storing one of (i) the configuration data of the task, or (ii) other configuration data of the other task in a fetch queue based on a comparison of the first priority and the second priority; The method of claim 13 further comprising:

16. 12. The method of claim 11, further comprising providing another portion of the configuration to a data buffer of the neural processor circuit, the other portion of the configuration data programming the data buffer to broadcast work units of input data for the task to the neural engine circuit.

17. 17. The method of claim 16, further comprising providing a third portion of the configuration data to a buffer direct memory access (DMA) of the neural processor circuit coupled to the data buffer and an external memory, the third portion of the configuration data programming the buffer DMA to obtain tiles of the input data for the task from the external memory and store the tiles in the data buffer, the tiles comprising a plurality of work units.

18. the portion of the configuration data programs an input buffer circuit of the neural engine circuit to provide a portion of the input data for the task stored in the input buffer circuit to a multiply-add (MAD) circuit of the neural engine circuit; said at least a portion of said configuration data programs an output circuit of said neural engine circuit to provide output data from said MAD circuit to a data buffer of said neural processor circuit; The method of claim 11.

19. providing another portion of the configuration data to a kernel direct memory access (DMA) coupled to the memory external to the neural processor circuit and to the neural engine circuit, the other portion of the configuration data programming the kernel DMA to retrieve kernel data from the memory external to the neural processor circuit and to provide the kernel data to the neural engine circuit to perform the task. The method of claim 11.

20. 1. An integrated circuit (IC) system comprising a neural processor circuit, the neural processor circuit comprising: A neural engine circuit, a neural task manager circuit coupled to the neural engine circuit, the neural task manager circuit comprising: a task queue circuit configured to store a reference to a task list of tasks for instantiating a neural network, the task list including configuration data for tasks stored in a memory location external to the neural processor circuit; a task arbiter circuit coupled to the task queue circuit, the task arbiter circuit comprising: retrieving the configuration data for the task from the location in the memory external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; a task arbiter circuit configured to provide at least a portion of the configuration data to the neural engine circuit, the at least a portion of the configuration data programming the neural engine circuit to perform the task; Including, Integrated circuit (IC) systems.

Images