System and method for assigning tasks in a neural network processor

Abstract

To provide a system and a method for enabling task management that instantiates a neural network in a neural processor circuit.SOLUTION: In a neural processor circuit including neural engine circuits and a neural task manager circuit, the neural task manager includes multiple task queues and a task arbiter. Each task queue stores a reference to a task list of tasks for a machine learning operation, and is associated with a priority parameter. On the basis of the priority of the task queues, the task arbiter retrieves configuration data 1014 for a task from a memory external to the neural processor circuit, and provides the configuration data to components of the neural processor circuit including the neural engine circuits. The configuration data programs the neural processor circuit to execute the task.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] 1. Disclosure in the technical field

[0002] This disclosure relates to circuits for implementing neural networks, and more specifically, to circuits for managing neural network tasks. [Background technology]

[0003] 2. Description of related technologies

[0004] An artificial neural network (ANN) is a computing system or model that uses a set of connected nodes to process input data. ANNs are typically organized into layers, with different layers performing different types of transformations on their inputs. Extensions or variations of ANNs, such as convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs), have gained significant attention. These computing systems or models often involve large-scale computations, including multiplication and accumulation. For example, a CNN is a class of machine learning techniques that primarily uses convolution between input data and kernel data, which can be broken down into multiplication and accumulation operations.

[0005] Depending on the type of input data and the operations performed, these machine learning systems or models can have different configurations. Such various configurations might include, for example, pre-processing operations, the number of channels in the input data, the kernel data used, the nonlinear functions applied to the convolution results, and the application of various post-processing operations. It is relatively easy to instantiate and run machine learning systems or models with various configurations using a central processing unit (CPU) and its main memory, because such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for the various operations of these machine learning systems or models significantly consumes CPU bandwidth and increases overall power consumption. [Overview of the project]

[0006] The embodiment relates to managing the task of instantiating a neural network when executed by a neural processor circuit. The neural processor circuit includes one or more neural engine circuits and a neural task manager circuit. The neural task manager circuit includes a plurality of task queue circuits and a task arbiter circuit. Each task queue circuit stores a reference to a task list of tasks for machine learning operations. Each task list and task queue circuit may be associated with a priority parameter. Based on the priority of the task queue circuits, the task arbiter circuit retrieves configuration data for the task from external memory and provides the configuration data to the components of the neural processor circuit, including one or more neural engines. The configuration data programs the neural processor circuit to execute the task.

[0007] Some embodiments include a method for managing tasks in a neural processor circuit. A reference to a task list of tasks that instantiate a neural network is stored in a task queue circuit of the neural processor circuit. The task list includes task configuration data stored in an external memory location. The task list is retrieved from the task queue circuit, for example, based on task priority. In response to retrieving the task list, the configuration data is retrieved from the external memory location. At least a portion of the configuration data is provided to the neural engine of the neural processor circuit to program the neural engine to execute the task. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram of an electronic device according to one embodiment. [Figure 2] This is a block diagram showing the components of an electronic device according to one embodiment. [Figure 3] A block diagram showing a neural processor circuit according to one embodiment. [Figure 4] This is a block diagram of a neural engine in a neural processor circuit according to one embodiment. [Figure 5] This is a conceptual diagram showing a loop for processing input data in a neural processor circuit according to one embodiment. [Figure 6] This is a conceptual diagram illustrating the segmentation of input data into slices, tiles, and work units according to one embodiment. [Figure 7] This figure shows the programming of a rasterizer within a component of a neural processor circuit according to one embodiment. [Figure 8] This is a flowchart showing a method for processing input data in a neural processor circuit according to one embodiment. [Figure 9] This is a schematic block diagram showing a neural network represented by a list of tasks, according to one embodiment. [Figure 10]This is a block diagram of a neural task manager according to one embodiment. [Figure 11] This figure shows the acquisition of a task descriptor using a task queue according to one embodiment. [Figure 12] This figure shows a task descriptor according to one embodiment. [Figure 13] This is a block diagram showing a fetch queue and a configuration queue for a neural task manager according to one embodiment. [Figure 14] This is a flowchart showing a method for managing tasks in a neural processor circuit according to one embodiment.

[0009] Various non-limiting embodiments are shown in the figures for illustrative purposes only and described in the detailed description. [Modes for carrying out the invention]

[0010] Herein, detailed references are made to embodiments shown in the accompanying drawings. The following detailed description includes numerous specific details to provide a complete understanding of the various embodiments described. However, the embodiments described can be carried out without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the aspects of the embodiments.

[0011] Embodiments of this disclosure relate to managing tasks that instantiate a neural network when executed by a neural processor circuit. Machine learning operations, such as inference or training operations, are defined by a task list of tasks. The neural processor circuit includes one or more neural engines and a neural task manager. The neural task manager includes a plurality of task queues and task arbiters. Each task queue stores a task list of tasks for machine learning operations. Each task list or task queue may be associated with a priority parameter. The task arbiter retrieves configuration data about a task from external memory based on the priority parameter and provides the configuration data to the components of the neural processor circuit, including one or more neural engines. In some embodiments, the neural task manager includes a configuration queue that stores configuration data for committed tasks selected by the task arbiter and provides the configuration data to other components of the neural processor circuit. The configuration data programs the neural processor circuit to execute the task. For example, the configuration data may include input data and kernel data that are processed by the neural engine to execute the task. The configuration data may further include instructions for acquiring and processing the configuration data, as well as instructions for storing the output data of the neural engine. In particular, the neural task manager enables the neural processor circuit to efficiently handle multiple machine learning operations. Furthermore, the neural task manager can facilitate task switching when a lower-priority task is being executed while a higher-priority task is stored in the task queue.

[0012] As used herein, a "task" refers to a processing operation of a neural processor circuit that instantiates a network layer of a neural network, a plurality of network layers of a neural network, or a portion of a network layer of a neural network. As used herein, a "task list" refers to a sequence of tasks, such as a sequence of tasks that are executed by a neural processor circuit to instantiate a plurality of network layers of a neural network. Exemplary Electronic Device

[0013] Embodiments of an electronic device, a user interface for such a device, and related processes for using such a device are described. In some embodiments, the device is a portable communication device, such as a cellular phone, that also includes other functions, such as a personal digital assistant (PDA) function and / or a music player function. Exemplary embodiments of a portable multifunctional device include, but are not limited to, devices such as the iPhone®, iPod Touch®, Apple Watch®, and iPad® from Apple Inc. of Cupertino, California. Other portable electronic devices, such as wearable computers, laptop computers, or tablet computers, may optionally be used. In some embodiments, the device is not a portable communication device, but is a desktop computer or other computing device not designed for portable use. In some embodiments, the electronic device of the present disclosure can include a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). The exemplary electronic device (e.g., device 100) described below in connection with FIG. 1 can include a touch-sensitive surface for receiving user input. The electronic device can also include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0014] FIG. 1 is a schematic diagram of an electronic device 100 according to an embodiment. The device 100 may include one or more physical buttons, such as a "home" or menu button 104. The menu button 104 is used, for example, to navigate to any application within a set of applications executed on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that identifies a fingerprint on the menu button 104. The fingerprint sensor can be used to determine whether a finger on the menu button 104 has a fingerprint that matches a fingerprint stored to unlock the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key within a graphical user interface (GUI) displayed on the touch screen.

[0015] In some embodiments, device 100 includes a touchscreen 150, a menu button 104, a push button 106 for turning the device on / off and locking the device, volume control buttons 108, a Subscriber Identity Module (SIM) card slot 110, a headset jack 112, and an external docking / charging port 124. The push button 106 may be used to turn the device on / off by pressing down the button and holding it down for a predetermined time interval, to lock the device by pressing down the button and releasing it before a predetermined time has elapsed, and / or to unlock the device or initiate an unlocking process. In alternative embodiments, device 100 also accepts verbal input via a microphone 113 to activate or deactivate certain functions. Device 100 includes a variety of components, including but not limited to memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral device interfaces, RF circuits, audio circuits, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may also include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may also include components not shown in Figure 1.

[0016] Device 100 is merely one embodiment of an electronic device, and device 100 may have more or fewer components than those listed above, and some of those components may be combined into a single component or have different configurations or arrangements. The various components of device 100 listed above are embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application-specific integrated circuits (ASICs).

[0017] Figure 2 is a block diagram showing the components of device 100 according to one embodiment. Device 100 can perform various operations, including image processing. For this purpose and other purposes, device 100 may include, among other components, an image sensor 202, a system-on-a-chip (SOC) component 204, a system memory 230, a persistent memory device (e.g., flash memory) 228, a motion (direction) sensor 234, and a display 216. The components shown in Figure 2 are merely illustrative. For example, device 100 may include other components not shown in Figure 2 (such as a speaker or microphone). Furthermore, some components (such as the motion sensor 234) may be omitted from device 100.

[0018] The image sensor 202 is a component for capturing image data and can be embodied, for example, as a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, a camera, a video camera, or other device. The image sensor 202 generates raw image data that is sent to the SOC component 204 for further processing. In some embodiments, the image data processed by the SOC component 204 is displayed on the display 216, stored in the system memory 230, the persistent storage device 228, or transmitted to a remote computing device via a network connection. The raw image data generated by the image sensor 202 can be a Bayer color kernel array (CFA) pattern (hereinafter also referred to as a "Bayer pattern").

[0019] The motion sensor 234 is a component or set of components for sensing the movement of device 100. The motion sensor 234 can generate sensor signals indicating the orientation and / or acceleration of device 100. The sensor signals are sent to the SOC component 204 for various actions, such as turning on device 100 or rotating an image displayed on display 216.

[0020] The display 216 is a component for displaying images such as those generated by the SOC component 204. The display 216 may include, for example, a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) device. Based on the data received from the SOC component 204, the display 216 can display various images, such as menus, selected operating parameters, images captured by the image sensor 202 and processed by the SOC component 204, and / or other information received from the user interface (not shown) of device 100.

[0021] The system memory 230 is a component for storing instructions to be executed by the SOC component 204 and for storing data processed by the SOC component 204. The system memory 230 can be embodied as any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, the system memory 230 may store pixel data or other image data, or statistics in various formats.

[0022] The persistent storage device 228 is a component for storing data non-volatilely. The persistent storage device 228 retains data even when power is unavailable. The persistent storage device 228 can be implemented as read-only memory (ROM), flash memory, or other non-volatile random access memory devices.

[0023] The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. Among other subcomponents, the SOC component 204 may include, in particular, an image signal processor (ISP) 206, a central processor unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and a bus 232 connecting these subcomponents. The SOC component 204 may include more or fewer subcomponents than those shown in Figure 2.

[0024] ISP206 is hardware that performs various stages of the image processing pipeline. In some embodiments, ISP206 can receive raw image data from the image sensor 202 and process the raw image data into a format usable by other subcomponents of the SOC component 204 or components of the device 100. ISP206 can perform various image manipulation operations, such as image transformation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as will be described in detail below with reference to Figure 3.

[0025] The CPU 208 may be implemented using any preferred instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. The CPU 208 may be a general-purpose or embedded processor using any of various ISAs, such as x86, PowerPC, SPARC, RISC, ARM, or MIPS instruction set architectures (ISAs), or any other preferred ISA. Although a single CPU is shown in Figure 2, the SOC component 204 may contain multiple CPUs. In a multiprocessor system, each CPU may, but not necessarily, implement the same ISA in common.

[0026] The graphics processing unit (GPU) 220 is a graphics processing circuit for executing graphics data. For example, the GPU 220 can render objects (e.g., those containing pixel data for the entire frame) that will be displayed in the frame buffer. The GPU 220 may include one or more graphics processors that can perform graphics software or hardware acceleration of specific graphics operations to perform some or all of the graphics operations.

[0027] The neural processor circuit 218 is a circuit that performs various machine learning operations based on calculations including multiplication, addition, and accumulation. Such calculations may be configured to perform, for example, convolution of input data and kernel data. The neural processor circuit 218 is a configurable circuit that performs these operations in a fast and power-efficient manner while freeing the CPU 208 from resource-intensive operations associated with neural network operations. The neural processor circuit 218 may receive input data from the sensor interface 212, the image signal processor 206, the system memory 230, or other sources such as the network interface 210 or the GPU 220. The output of the neural processor circuit 218 may be provided to various components of device 100, such as the image signal processor 206, the system memory 230, or the CPU 208, for various operations. The structure and operation of the neural processor circuit 218 are described in detail below with reference to Figure 3.

[0028] The network interface 210 is a subcomponent that enables the exchange of data between device 100 and other devices over one or more networks (e.g., carrier or agent devices). For example, video or other image data may be received from other devices via the network interface 210 and stored in system memory 230 for subsequent processing and display (e.g., via a backend interface to the image signal processor 206, as described later with respect to Figure 3). The network may be, but is not limited to, a local area network (LAN) (e.g., Ethernet or a corporate network) and a wide area network (WAN). Image data received via the network interface 210 can be subjected to an image processing process by the ISP 206.

[0029] The sensor interface 212 is a circuit for interfacing with the motion sensor 234. The sensor interface 212 receives sensor information from the motion sensor 234, processes this sensor information, and determines the orientation or movement of the device 100.

[0030] The display controller 214 is a circuit for transmitting image data that will be displayed on the display 216. The display controller 214 receives image data from the ISP 206, CPU 208, graphics processor, or system memory 230, and processes the image data into a format suitable for display on the display 216.

[0031] The memory controller 222 is a circuit for communicating with the system memory 230. The memory controller 222 can read data from the system memory 230 for processing by the ISP 206, CPU 208, GPU 220, or other subcomponents of the SOC component 204. The memory controller 222 can also write data received from various subcomponents of the SOC component 204 to the system memory 230.

[0032] The video encoder 224 is hardware, software, firmware, or a combination thereof for encoding video data into a format suitable for storage in the persistent memory device 128, or for passing data to the network interface 210 for transmission to another device over a network.

[0033] In some embodiments, one or more subcomponents of the SOC component 204, or some functions of these subcomponents, can be performed by software components running on the ISP 206, CPU 208, or GPU 220. Such software components can be stored in system memory 230, persistent storage 228, or another device that communicates with device 100 via the network interface 210.

[0034] Image data or video data can flow through various data paths within the SOC component 204. In one embodiment, raw image data can be generated from the image sensor 202, processed by the ISP 206, and then transmitted to the system memory 230 via the bus 232 and the memory controller 222. After the image data is stored in the system memory 230, it can be accessed via the bus 232 by the video encoder 224 for encoding or by the display 116 for display. Exemplary Neural Processor Circuit

[0035] The neural processor circuit 218 is a configurable circuit that performs neural network operations on input data based on at least kernel data 340. For this purpose, the neural processor circuit 218 may include, among other components, a neural task manager 310, a plurality of neural engines 314A to 314N (hereinafter collectively referred to as "neural engine 314" and individually referred to as "neural engine 314"), a kernel direct memory access (DMA) 324, a data buffer 318, and a buffer DMA 320. The neural processor circuit 218 may also include other components not shown in Figure 3.

[0036] Each of the neural engines 314 performs computational operations for neural network operation in parallel. Depending on the workload, the entire set of neural engines 314 may be operated, or only a subset of the neural engines 314 may be operated while the remaining neural engines 314 are placed in power-saving mode to conserve power. Each of the neural engines 314 includes components for storing one or more kernels, for performing multiplication-cumulative operations, and for post-processing to generate output data 328, as will be described in detail below with reference to Figure 4. An example of neural network operation is a convolution operation.

[0037] The neural task manager 310 manages the overall operation of the neural processor circuit 218. The neural task manager 310 may receive a task list from a compiler run by the CPU 208, store tasks in its task queue, select a task to execute, and send instructions to other components of the neural processor circuit 218 to execute the selected task. The neural task manager 310 may also perform task switching when it detects an event, such as receiving an instruction from the CPU 208. In one or more embodiments, as described in detail below with reference to Figures 5 to 7, the neural task manager 310 sends rasterizer information to components of the neural processor circuit 218, enabling each component to track, retrieve, or process the appropriate portions of the input data and kernel data. Although the neural task manager 310 is shown as part of the neural processor circuit 218 in Figure 3, the neural task manager 310 may be a component located outside of the neural processor circuit 218.

[0038] The kernel DMA324 is a readout circuit that fetches kernel data from a source (e.g., system memory 230) and sends kernel data 326A to 326N to each of the neural engines 314. The kernel data represents information from which kernel elements can be extracted. In one embodiment, the kernel data may be in a compressed format compressed by each of the neural engines 314. The kernel data provided to each of the neural engines 314 may be the same in some cases, but in most cases the kernel data provided to each of the neural engines 314 is different.

[0039] The data buffer 318 is a temporary storage device for storing data associated with the operation of the neural network. In one embodiment, the data buffer 318 is embodied as a memory accessible by all of the neural engines 314. The data buffer 318 may store input data 322A to 322N for supply to the corresponding neural engines 314A to 314N, as well as outputs from each of the neural engines 314A to 314N for feedback to the neural engines 314 or for transmission to a target circuit (e.g., system memory 230). The operation of the data buffer 318 and other components of the neural processor circuit 218 is coordinated so that the input and intermediate data stored in the data buffer 318 are reused across multiple operations in the neural engines 314, thereby reducing data transfer to and from the system memory 230. The data buffer 318 may operate in broadcast mode, where input data from all input channels is supplied to all neural engines 314, or in unicast mode, where input data from a subset of input channels is supplied to each neural engine 314.

[0040] The input data 322 stored in the data buffer 318 may, among other things, be part of the image data from previous cycles of the neural engine 314, luminance gradient histogram (HOG) data, audio data, metadata, output data 328, and other processed data received from other components of the SOC component 204.

[0041] The buffer DMA320 includes a read circuit that receives a portion (e.g., tiles) of input data from a source (e.g., system memory 230) for storage in the data buffer 318, and a write circuit that delivers data from the data buffer 318 to a target (e.g., system memory). Exemplary neural engine architecture

[0042] Figure 4 is a block diagram of a neural engine 314 according to one embodiment. The neural engine 314 performs various operations that facilitate neural network operation, such as convolution, spatial pooling, and local response normalization. The neural engine 314 receives input data 322, performs a multiplication-cumulative operation (e.g., a convolution operation) on the input data 322 based on stored kernel data, performs further post-processing operations on the result of the multiplication-cumulative operation, and generates output data 328. The input data 322 and / or output data 328 of the neural engine 314 may be single-channel or multi-channel.

[0043] The neural engine 314 may include, among other components, an input buffer circuit 402, a computing core 416, a neural engine (NE) control unit 418, a kernel extraction circuit 432, a accumulator 414, and an output circuit 424. The neural engine 314 may also include further components not shown in Figure 4.

[0044] The input buffer circuit 402 is a circuit that receives input data 322 from the data buffer 318, stores a portion of it, and sends it to the computing core 416 for processing the portion of input data 408 appropriate for the current task or process loop. The input buffer circuit 402 includes a shifter 410 that shifts the read location of the input buffer circuit 402, thereby modifying the portion of input data 408 sent to the computing core 416. By modifying the portion of input data provided to the computing core 416 via the shift, the neural engine 314 can perform multiplication-accumulation on different portions of the input data based on fewer read operations. In one or more embodiments, the input data 322 includes difference convolution groups and / or input channel data.

[0045] The kernel extraction circuit 432 is a circuit that receives kernel data 326 from the kernel DMA 324 and extracts kernel coefficients 422. In one embodiment, the kernel extraction circuit 432 refers to a look-up table (LUT) and uses a mask to reconstruct the kernel from the compressed kernel data 326. The mask indicates where in the reconstructed kernel to be filled with zeros, and the remaining places to be filled with numbers. The kernel coefficients 422 of the reconstructed kernel are sent to the compute core 416 and read into registers in the compute core 416's multiply-add (MAD) circuit. In other embodiments, the kernel extraction circuit 432 receives the kernel data in uncompressed form, and the kernel coefficients are determined without referring to the LUT or using a mask.

[0046] The computing core 416 is a programmable circuit that performs computational operations. For this purpose, the computing core 416 may include MAD circuits MAD0 to MADN and a post-processor 428. Each of the MAD circuits MAD0 to MADN may store input values ​​in input data section 408 and corresponding kernel coefficients in kernel coefficients 422. The input values ​​and corresponding kernel coefficients are multiplied in each MAD circuit to generate a processed value 412.

[0047] The accumulator 414 is a memory circuit that receives and stores the processed value 412 from the MAD circuit. The processed value stored in the accumulator 414 may be transmitted as feedback information 419 for further multiplication and addition operations in the MAD circuit, or it may be transmitted to the post-processor 428 for post-processing. The accumulator 414, in combination with the MAD circuit, forms a multiply-accumulator (MAC) 404. In one or more embodiments, the accumulator 414 may have subunits, each subunit transmitting data to a different component of the neural engine 314. For example, during a processing cycle, data stored in the first subunit of the accumulator 414 is transmitted to the MAC circuit, and data stored in the second subunit of the accumulator 414 is transmitted to the post-processor 428.

[0048] The post-processor 428 is a circuit that performs further processing on the value 412 received from the accumulator 414. The post-processor 428 may perform operations including, but are not limited to, nonlinear functions (e.g., Rectified Linear Unit, ReLU), normalized cross-correlation (NCC), merging the results of neural operations performed on 8-bit data into 16-bit data, and local response normalization (LRN). The results of such operations are output from the post-processor 428 to the output circuit 424 as the processed value 417.

[0049] The NE control unit 418 controls the operation of other components of the neural engine 314 based on the operating mode and parameters of the neural processor circuit 218. Depending on different operating modes (e.g., group convolution mode or non-group convolution mode) or parameters (e.g., the number of input channels and the number of output channels), the neural engine 314 may operate in different sequences for different input data, return different values ​​from the accumulator 414 to the MAC circuit, and perform different types of post-processing operations in the post-processor 428. To configure the components of the neural engine 314 to operate in a desired manner, the NE control unit 418 transmits control signals to the components of the neural engine. The NE control unit 418 may also include a rasterizer 430 that tracks the current task or process loop being processed by the neural engine 314, as will be described in detail later with reference to Figures 5 to 7.

[0050] The output circuit 424 receives the processed value 417 from the post-processor 428, interfaces with the data buffer 318, and stores the processed value 417 in the data buffer 318. For this purpose, the output circuit 424 may send out the output data 328 in order, in a non-order format, or in a format in which the processed value 417 is processed by the post-processor 428.

[0051] The components within the neural engine 314 may be configured by the NE control unit 418 and the neural task manager 310 during the configuration period. For this purpose, the neural task manager 310 transmits configuration information to the neural engine 314 during the configuration period. Configurable parameters and modes include, but are not limited to, the mapping between input data elements and kernel elements, the number of input channels, the number of output channels, the execution of output stride, and the enablement / selection of post-processing operations in the post-processor 428. The operation of data segmentation for processing in neural processor circuits.

[0052] The input data is typically divided into smaller data pieces for parallel processing across multiple neural engines 314. Often, multiple cycles of operation are performed to generate the output of the task associated with the neural network. The compiler, run by the CPU 208, analyzes the hierarchy and nodes of the neural network and determines how the input data should be segmented based on the hardware constraints of the neural processor circuit 218. One of the compiler's functions is to determine how the input data should be divided into smaller data units for processing in the neural engines 314, and how the processing should be repeated in a loop to generate results for the task.

[0053] Figure 5 is a conceptual diagram showing a loop for processing input data in a neural processor circuit 218 according to one embodiment. The outermost loop represents the processing of convolution groups when group convolution with multiple convolution groups is used. Group convolution is a convolution in which the input data of the input channels within each group is used only to generate the output data of the output channels of that group, but not to generate the output data for the output channels of other groups. Therefore, each group in a group convolution can be treated as a separate convolution operation.

[0054] Each convolution group's loop contains a processing loop for slices of input data. The entire input data for the convolution operation is segmented into multiple strips of slices in an overlapping form, as shown in Figure 6. Overlapping portions 602, 604, and 606 are portions of input data that have been overfetched in two adjacent slices to provide spatial support for the corresponding kernels. The second loop from the outside performs the convolution operation for each slice in the input data. Within the loop for slices, there is a processing loop for tiles of the slices. Each slice is segmented into multiple tiles, as shown in Figure 6. Overlapping portions 608, 610, 612, and 614 are portions of input data in slice 4 that have been overfetched in two adjacent tiles to provide spatial support for the corresponding kernels. The rightmost tile will typically have a smaller width than the other tiles in the slice. In one embodiment, the input data for each tile is loaded onto a data buffer 318 during a read cycle and reused for operations in the processing loop for the tile. Within the processing loop for tiles, there is a processing loop for work units. Each tile is segmented into multiple work units, as shown in Figure 6. A work unit is a portion of the input data having a size that produces an output value that fits into the accumulator 414 of the neural engine 314 during a single cycle of the computing core 416. The shape of each work unit is shown as a horizontal strip in Figure 6, but the shape of the work unit may vary depending on the shape and size of the tile. A work unit also has overlapping portions that represent overfetched data to provide support for the corresponding kernel. In particular, the work unit of the last tile in a slice may have the shape of a vertical strip if the tile is elongated. In one or more embodiments, the size of each work unit is 256 bytes. In such embodiments, for example, a work unit may be shaped to one of the dimensions of 16×16, 32×8, 64×4, 128×2, or 256×1.

[0055] An internal processing loop for an output channel group (OCG) may be provided for each work unit. The number of output channels generated for a given work unit in a single cycle of the computing core 416 is referred to as the OCG. Depending on the operating mode, each neural engine 314 may process output data for a different number of output channels (e.g., 8 channels, 32 channels) for a single load of input data into the input buffer circuit 402.

[0056] For each output channel group, an internal processing loop for the input channel (Cin) may be provided. If the input stride is implemented to skip specific input data, the loop for the sub-input channel (Sub-Cin) may be provided within the processing loop for the input channel (Cin).

[0057] For each input channel or each sub-input channel, an internal loop is provided to handle the horizontal spatial support to the kernel and the vertical support within each horizontal spatial support. The spatial support refers to the input data for convolution with the kernel and includes overfetched input data at the ends of the input data for performing the convolution.

[0058] Overfetching refers to fetching additional input data within the current slice, tile, or work unit so that the appropriate dimensions of the input data can be provided for convolution with the kernel. In one or more embodiments, overfetching is performed vertically to obtain additional input data rows between slices (shown as overlapping portions 602, 604, and 606 in Figure 6), horizontally to obtain additional input data columns between tiles (shown as overlapping portions 608, 610, 612, and 614 in Figure 6), and vertically to obtain additional input data rows between work units within a tile.

[0059] For each spatial support for the kernel, an internal processing loop is provided for the output channel (OC), and output data is generated for each output channel (Cout). If the output stride implements spatial upsampling, an additional inner loop is provided to process each sub-output channel. When the output stride is implemented, the kernel coefficients and MAC operation are loaded within the loop for the output channel (OC) or sub-output channel, and output data is generated for the output channel (OC) or sub-output channel.

[0060] The nested loop structure in Figure 5 is merely illustrative. Loops may be omitted, added, or structured depending on various factors. For example, if only a single convolution group is used, the outermost loop may be removed. Furthermore, the loop structures for horizontal and vertical spatial support may be inverted.

[0061] In one or more embodiments, as described above with reference to Figures 5 and 6, the operations associated with dividing the input space into smaller units and processing these smaller units are performed by rasterizers 714, 718, 720, and 722 within various components of the neural processor circuit 218. Rasterizers are circuits within various components of the neural processor circuit 218 that track segments of input / output data (e.g., groups, work units, input channels, output channels) and instruct components of the neural processor circuit for proper handling of the input data segments. For example, rasterizer 720 in buffer DMA 320 tracks tiles and slices received from system memory 230, while rasterizer 718 in data buffer 318 broadcasts work units in order for processing by the neural engine 314. The rasterizer 722 within the kernel DMA 322 determines which kernel is received and distributed to the neural engine 314, while the rasterizer 714 within the neural engine 314 operates the shifter 410 within the input buffer circuit 402 to deliver the correct portion 408 of the input data to the MAC 404 and send the final output data 328 to the data buffer 318.

[0062] Figure 7 shows the programming of rasterizers 714, 718, 720, and 722 within components 314, 318, 320, and 322 of a neural processor circuit 218 according to one embodiment. To perform their functions, each of the rasterizers 714, 718, 720, and 722 receives task information 710 (e.g., configuration data) indicating how the input data and / or kernel data should be segmented and handled by each component of the neural processor circuit 218. The task information includes information about the details of the current layer (e.g., dimensions of the input and output data, dimensions of the associated kernel, type of padding at the boundaries of the input data). The rasterizers 714, 718, 720, and 722 may also receive constraints on their operation (e.g., whether to allow or not allow tile widths exceeding a threshold).

[0063] By providing rasterizers within different components of the neural processor circuit 218, the overhead in data transmitted between components of the neural processor circuit 218 can be reduced. If a single central rasterizer is provided to control the different components of the neural processor circuit 218, this data may require kernel data, input data, and output data to be transmitted between components to identify associated locations within the task loop, such as convolutional groups, tiles, slices, work units, input channels, and output channels. By using distributed rasterizers, separate metadata is not required to transmit kernel data, input data, and output data between components of the neural processor circuit 218. Exemplary processes in neural engine architecture

[0064] Figure 8 is a flowchart showing a method for processing input data in a neural processor circuit 218 according to one embodiment. This method may include different and / or additional steps, or the steps may be in a different order.

[0065] After the neural task manager 310 has programmed the rasterizers 714, 718, 720, and 722, the process of operating the buffer DMA 320 is initiated when the rasterizer 720 instructs the buffer DMA 320 to receive tiles of input data from the system memory 230 (804). The tiles received by the buffer DMA 320 are stored in the data buffer 318 (806).

[0066] The rasterizer 718 in the data buffer 318 then instructs the data buffer 318 to send a work unit to one or more neural engines 314 (808). The work unit is then stored in the input buffer circuit 402 of one or more neural engines 314.

[0067] In one or more embodiments, the input buffer circuit 402 selects a portion of the work unit to be sent to the MAC 404 for multiplication-cumulative operation (816). The MAC 404 then performs the multiplication-cumulative operation on the selected portion of the work unit using the corresponding kernel (820). It then determines whether the entire work unit is being processed by one or more neural engines 314 (824). If the determination is no, the selected portion of the work unit is shifted by the shifter 410 (828) and the multiplication-cumulative operation is performed again (820).

[0068] If it is determined that the entire work unit has been processed (824), the process proceeds to determine whether all work units in the tile have been processed (832). If the determination is no, the process proceeds to the next work unit (836) by causing the data buffer 318 to send the next work unit to one or more neural engines 314 (808), and the subsequent process is repeated.

[0069] If it is determined that all work units within a tile have been processed by the neural engine 314 (832), the process proceeds to determine whether all tiles of the input data have been processed (840). If the determination is no, the process proceeds to the next tile (844) by instructing the rasterizer 720 to receive the next tile from the system memory 230 by instructing the buffer DMA 320 (804), and the subsequent process is repeated.

[0070] If it is determined that all tiles of the input data have been processed (840), the process for the current input data ends. The process may then be repeated to process the next input data, or it may proceed to the next task.

[0071] The process embodiments described above with reference to Figure 8 are merely illustrative. Further loops may be implemented, as described above with reference to Figure 5. Moreover, the process sequence may be modified or omitted. Exemplary Neural Task Manager Architecture

[0072] A neural network may include network layers or sublayers that are instantiated or implemented as a series of tasks performed by a neural processor circuit 218. Figure 9 is a schematic block diagram showing a neural network 900 represented by a list of tasks 904 according to one embodiment. The neural network 900 includes convolutional layers C1, C2, C3 (sublayer C3 00 , C3 10 , C3 11 , C3 20 , and C3 21including), C5, and C6, and network layers (or sub - layers) including pooling layers P2 and P4. Neural network 900 is an example of a neural network architecture that can be instantiated by neural processor circuit 218. That is, when neural network 900 is converted into task list 904 and becomes executable by neural processor circuit 218. Other types of neural network architectures having different types of network layers or different orders of network layers may also be instantiated by neural processor circuit 218.

[0073] Neural network 900 is converted into task list 904, for example, by CPU 208. Task list 904 defines a linear linked list including a sequence of tasks including task C1, task C2 + P2, task C3 00 + P4, task C3 10 task C3 11 + P4, task C3 20 task C3 21 + P4, task C5 a task C5 b and task C6. Each task is associated with a task descriptor that defines the configuration of neural processor circuit 218 for executing the task. Each task may correspond to a single network layer of neural network 900, a part of the network layers of neural network 900, or multiple network layers of neural network 900. For example, task C1 corresponds to a single network layer C1, task C2 + P2 corresponds to multiple network layers C2 and P2, and task C5 a and C5 b each correspond to a part of network layer C5. Neural processor circuit 218 instantiates neural network 900 by executing the tasks in task list 904 under the control of neural task manager 310.

[0074] Figure 10 is a block diagram of a neural task manager 310 according to one embodiment. The neural task manager 310 manages the execution of tasks for one or more neural networks 900 by a neural processor circuit 218. The neural task manager 310 may include, among other components, a task arbiter 1002, task queues 1004A to 1004N (hereinafter collectively referred to as "task queue 1004" and also individually referred to as "task queue 1004"), a task manager direct memory access (DMA) 1006, a fetch queue 1008, and a configuration queue 1010. The neural task manager 310 may include other components not shown in Figure 10.

[0075] The task arbiter 1002 is either a circuit executed by the neural processor circuit 218 that selects a task from the task queue 1004, or a combination of the circuit and firmware. The task arbiter 1002 dequeues a task from the task queue 1004 and places the task in the configuration queue 1010. While the task is in the configuration queue, its execution is committed, and the neural processor circuit performs prefetching of input data and kernel data before the task is executed by other components of the neural processor circuit 218. For example, the task arbiter 1002 may perform fixed-priority arbitration among multiple task queues 1004 and select the task with the highest priority from the task queue 1004 for the task manager DMA 1006 to retrieve the task descriptor 1012 from the system memory 230.

[0076] The neural task manager 310 may include one or more task queues 1004. Each task queue 1004 is coupled to the CPU 208 and the task arbiter 1002. Each task queue 1004, when executed by the neural processor circuit 218, receives a reference from the CPU 208 to a task list 904 of tasks that instantiate the neural network 900. The reference stored in each task queue 1004 may include a set of pointers and counters pointing to the task list 904 of a task descriptor 1012 in system memory 230. Each task queue 1004 may further be associated with priority parameters that define the relative priority of the task queue 1004. The task descriptor 1012 of a task specifies the configuration of the neural processor circuit 218 for executing the task.

[0077] The task manager DMA 1006 is coupled to the task arbiter 1002, system memory 230, and fetch queue 1008. The task manager DMA 1006 includes a read circuit that receives task descriptors 1012 of tasks from a source (e.g., system memory 230) for storage in the fetch queue 1008. For example, the task arbiter 1002 selects a task queue 1004 according to its priority and uses the task list 904 referenced by the selected task queue 1004 to control the task manager DMA 1006 to select task descriptors 1012 of tasks.

[0078] The fetch queue 1008 is a single-entry queue that stores task descriptors 1012 for tasks whose commits to be executed are pending. The fetch queue 1008 is coupled to the task manager DMA 1006 to receive task descriptors 1012 from system memory 230 and provide the task descriptors 1012 to the configuration queue 1010, or provide the configuration data 1014 extracted from the task descriptors 1012 to the configuration queue 1010.

[0079] The configuration queue 1010 holds configuration data 1014 for multiple tasks whose execution has been committed. When a task is in the configuration queue 1010, the kernel DMA 324 may fetch kernel data from system memory 230 and store it in the kernel extraction circuit 432 of the neural engine 314, and the buffer DMA 320 may fetch input data from system memory 230 and store it in the data buffer 318. To execute a task, the kernel extraction circuit 432 provides the prefetched kernel data to the MAC 404 of the neural engine 314, and the data buffer 318 provides the prefetched input data to the MAC 404 of the neural engine 314. In some embodiments, the configuration queue 1010 may include multiple queues that hold configuration data 1014 extracted from committed task descriptors 1012. As discussed in more detail with reference to Figure 13, the configuration queue 1010 is further coupled to other components of the neural processor circuit 218 to configure the neural processor circuit 218 according to the configuration data 1014.

[0080] Figure 11 shows the retrieval of a task descriptor 1012 using a task queue 1004 according to one embodiment. The task queue 1004 includes references, for example, a set of pointers, to task descriptors 1012A to 1012N stored in system memory 230. For this purpose, the task queue 1004 may include memory for storing a head parameter 1102, a network identifier (ID) 1104, a base address index 1106, a tail parameter 1108, a count parameter 1110, and a priority parameter 1112. The head parameter 1102 is a pointer to the location in system memory 230 that stores task descriptor 1012A, located at the head of the task queue 1004. Network ID 1104 identifies the neural network 900 of task descriptor 1012 at the head of task queue 1004, and base address index 1106 is the index to the base address table 1114 tagged with network ID 1104 for task descriptor 1012A at the head of task queue 1004. The count parameter 1110 defines the number of task descriptors 1012 in task queue 1004. The priority parameter 1112 defines the priority of task queue 1004, which task arbiter 1002 uses to select from among multiple task queues 1004.

[0081] If a specific task queue 1004 is selected (for example, according to the priority parameter 1112), the task arbiter 1002 retrieves the task descriptor 1012 from system memory 230 by referring to the head parameter 1102, network ID 1104, base address index 1106, and base address table 1114, places the task descriptor 1012 in the fetch queue 1008, and initiates a commitment to task execution. During each configuration period, the task arbiter 1002 may continue to place the task descriptor 1012 in the fetch queue 1008 according to the order of tasks defined by the task list 904 of task queue 1004, for example by retrieving the next task descriptor 1012B.

[0082] Figure 12 shows a task descriptor 1012 according to one embodiment. The task arbiter 1002 places the task descriptor 1012 from the system memory 230 into the fetch queue 1008, and the task descriptor 1012 is then transferred to the configuration queue 1010. The task descriptor 1012 with the highest priority in the configuration queue 1010 (e.g., the first to be entered) is used to configure the neural processor circuit 218 for execution during the configuration period. The task descriptor 1012 includes a task descriptor header 1202 and configuration data 1014 which includes address data 1204A to 1204N (hereinafter referred to as "address data 1204"). The task descriptor header 1202 includes configuration data 1014 which constitutes various operations of the neural task manager 310, including operations related to task selection and task switching. For example, the task descriptor header 1202 may be parsed by the task arbiter 1002 to extract configuration data 1014 for programming the neural task manager 310 and other components of the neural processing circuit 218. The task descriptor header 1202 may include a task identifier (ID) 1206 that identifies a task, a neural network identifier (ID) 1208 that identifies a neural network 900 instantiated by the task, a task switching parameter 1210 that defines whether the neural task manager 310 should initiate task switching after the task has been executed (for example, to execute a task in a different task queue 1004), an input surface parameter 1212 that defines whether input data for the task should be obtained from system memory 230 or data buffer 318, an output surface parameter 1214 that defines whether output data for the task should be stored in system memory 230 or data buffer 318, various (e.g., base address) pointers 1216 to facilitate programming of the neural processor circuit 218, and one or more debug / exception parameters 1218 that control events, exceptions, or debug logs.

[0083] Each instance of address data 1204A to 1204N (collectively or individually referred to as "address data 1204") defines an address-data payload pair used to program the components of the neural processor circuit 218. The data payload may include input data and kernel data used to execute a task. For example, each instance of address data 1204 includes register data defining the data payload, a register address defining the destination memory location of the neural processing circuit 218 for receiving the register data, and a register count defining a contiguous memory location (e.g., a register) to which the register data will be written. In some embodiments, the register address, combined with a base address stored in the header 1202, defines the full address of each memory location. If the task descriptor 1116 is generated at compile time, the actual runtime address may not be known. The base address table 1114 is used to prevent all task descriptors from being duplicated or updated with dynamically allocated addresses.

[0084] Figure 13 is a block diagram showing a fetch queue 1008 and a configuration queue 1010 according to one embodiment. The configuration queue 1010 is coupled to the fetch queue 1008, which is coupled to system memory 230 via a task manager DMA 1006. The configuration queue 1010 is further coupled to the rasterizers 714 of one or more neural engines 314, the rasterizer 718 of a data buffer 318, the rasterizer 720 of a buffer DMA 320, and the rasterizer 722 of a kernel DMA 322. The fetch queue 1008 stores task descriptors 1012 (e.g., including a task descriptor header 1202 and address data 1204A-1204N) for pending tasks whose execution has not been committed. The fetch queue 1008 reduces the latency of reading the next task descriptor 1012 from system memory 230 into the configuration queue 1010. The fetch queue 1008 stores the task descriptor 1012 with the highest priority, as determined by the task arbiter 1002. The task arbiter 1002 may replace the task descriptor 1012 stored in the fetch queue 1008 if a higher priority task descriptor 1012 is enqueued (for example, from a higher priority task queue 1004). The task descriptor 1012 in the fetch queue 1008 does not initiate input data or kernel prefetching and does not affect task queue priorities, pointers, or counters. Therefore, the task descriptor 1012 in the fetch queue 1008 may be easily replaced by a higher priority task descriptor 1012 by writing the higher priority task descriptor 1012 to the fetch queue 1008. If a task descriptor 1012 stored in the configuration queue 1010 is executed by the neural processor circuit 218, the task descriptor 1012 stored in the fetch queue 1008 may be transferred to the configuration queue 1010, and another task descriptor 1012 for a subsequent task may be stored in the fetch queue 1008.

[0085] The configuration queue 1010 stores task descriptors 1012 of tasks committed to execution by the neural processor circuit 218. In some embodiments, the configuration queue 1010 includes a plurality of separate queues 1310, each of which stores a portion of the configuration data 1014 (including configuration data 1014A to 1014E) extracted from the task descriptors 1012. Furthermore, each queue 1310 is coupled to a corresponding component of the neural processor circuit 218 and programmed with the configuration data 1014. Through the operation of the configuration queue 1010, the neural task manager 310 programs the rasterizers 714, 718, 720, and 722 to perform the functions described above in Figures 7 and 8. For example, queue 1310A is coupled to the rasterizer 714 of the neural engine 314 and provides configuration data 1014A that controls the operation of the shifter 410 in the input buffer circuit 402 to deliver the correct portion 408 of the input data to the MAC 404 and send the final output data 328 to the data buffer 318. Queue 1310B is coupled to the rasterizer 718 of the data buffer 318 and provides configuration data 1014B that controls the broadcasting of input data (e.g., work units) by the data buffer 318 for processing by the neural engine 314. Queue 1310C is a read queue coupled to the rasterizer 720 of the buffer DMA 320 and provides configuration data 1014C that controls the buffer DMA 320 to retrieve input data (e.g., tiles) from the system memory 230 and store the input data in the data buffer 318. Queue 1310D is a write queue coupled to the rasterizer 720 of the buffer DMA320 and provides configuration data 1014D that controls the buffer DMA320 and stores output data in the system memory 230. Queue 1310E is coupled to the rasterizer 722 of the kernel DMA322 and provides configuration data 1014E that controls which kernel is received and distributed to the neural engine 314. In some embodiments, task descriptors 1012 or configuration data 1014 stored in configuration queue 1010 cannot be replaced and are executed in a first-in, first-out (FIFO) order. Exemplary processes in the neural task manager architecture

[0086] Figure 14 is a flowchart showing a method for managing tasks in a neural processor circuit 218 according to one embodiment. This method may include different and / or additional steps, or the steps may be in a different order.

[0087] The CPU 208, when executed by the neural processor circuit 218, generates a task list 904 of task descriptors 1012 for the task of instantiating the neural network 900 (1402). For example, the CPU 208 may receive input data for machine learning operations from the image sensor 202, system memory 230, persistent storage device 228, network interface 210, or any other component. The machine learning operations may include inference operations or training operations. The neural network 900 may include a neural network architecture including kernel data and network layers. The input data is applied to the neural network 900 to perform the machine learning operations. The kernel data and network layers of the neural network 900 may be computed during the machine learning training process. The CPU 208 performs a compilation operation (offline or on the fly) to convert the neural network description 900 into a linked list of task descriptors 1012, referred to herein as the task list 904. Each task is defined by a task descriptor 1012, which, when executed by the neural processor circuit 218, instantiates a single network layer, multiple network layers, or a portion of network layers. Each task descriptor 1012 of a task includes configuration data 1014, such as a task descriptor header 1202, and address data 1204, which defines a neural processor circuit 218 address and data payload pair. The data payload may include kernel data for the neural network 900 and input data. The configuration data 1014 further includes instructions that constitute the operation of rasterizers 714, 718, 720, and 722 to execute the task.

[0088] The CPU 208 stores the task list 904 of the task descriptor 1012 in the system memory 230 (1404). In some embodiments, the CPU 208, or another CPU outside the electronic device 100, generates the task descriptor 1012 and stores the task descriptor 1012 in the persistent memory 228 or some other non-volatile memory. The task descriptor 1012 is loaded together with kernel data and input data in the system memory 230 for use by the neural processor circuit 218. The CPU 208 may be coupled to the system memory 230 via the bus 232 and the memory controller 222.

[0089] The task list 904 may include a set of pointers that reference the location of the task descriptor 1012 in system memory 230. Furthermore, the CPU 208 may update parameters of the task descriptor 1102, such as memory addresses or network IDs. For example, the task list 904 may include head parameters 1102 and tail parameters 1108 that define the start and end register addresses of the system memory 230 where multiple task descriptors 1012 are stored, respectively. In some embodiments, references to register addresses in the task list 904 may be partial addresses, and the base address table 1114 is used to define full reference addresses to the system memory 230. In some embodiments, the CPU 208 may patch absolute addresses as needed. The CPU 208 may further configure the neural processor circuit 218 by setting its base address registers.

[0090] The task arbiter 1002 of the neural task manager 310 enqueues a task list 904 to be executed by the neural processor circuit 218 (1406). For example, the neural task manager 310 includes multiple task queues 1004A to 1004N. Each task queue 1004 may store a reference to the task list 904. Furthermore, the task queues 1004 are prioritized to be executed according to the priority parameter 1112 of the task list 904 referenced by the task queue 1004.

[0091] In some embodiments, the CPU 208 executes a general configuration of the neural processor circuit 218 to perform a task. If the neural processor circuit 218 is not already running, the CPU 208 may further start the neural processor circuit 218.

[0092] The task arbiter 1002 selects the task queue with the highest priority for execution (1408). For example, during each programming period, the task arbiter 1002 selects a task queue 1004 based on a comparison of the priority parameters of the task queues 1004 or a comparison of the task list 904 of the task queue 1004, and executes tasks from the task list 904 starting with the task queue 1004 with the highest priority.

[0093] The neural processor circuit 218 executes a task from the selected task queue (1410). For example, the neural processor circuit 218 executes a task using the method shown in Figure 8. The configuration queue 1010 of the neural task manager 310 may provide configuration data 1014 (or task information 710) to the rasterizers 714, 718, 720, and 722 of the neural processor circuit 218, as shown in Figure 7, to program the neural processor circuit 218 to execute a task. Furthermore, task execution may include processing prefetched kernel data and input data while the task is in the configuration queue 1010. Each task execution may include multiple processing loops for handling input data, as shown in Figures 5 and 6. The task arbiter 1002 may dequeue the task from the configuration queue 1010 after the task has been executed.

[0094] In some embodiments, to execute a task, the task arbiter 1002 controls the task manager DMA 1006 to retrieve the task descriptor 1012 of the task in the task list 904 and stores the task descriptor 1012 in the fetch queue 1008. After the execution of a previously committed task, the task arbiter 1002 may dequeue the executed task by removing the task descriptor or configuration data of the task from the configuration queue 1010. The task descriptor 1012 or extracted configuration data 1014 of the current task is then placed from the fetch queue 1008 into the configuration queue 1010. If the task is in the configuration queue 1010, the neural processor circuit 218 may initiate a prefetch operation by the kernel DMA 324 for kernel data from the system memory 230 to the kernel extraction circuit 432, and a prefetch operation by the buffer DMA 320 for input data from the system memory 230 to the data buffer 318.

[0095] If different task queues have a higher priority, a task switch to the task queue with the higher priority may be performed if possible. To perform a task switch (for example, to a different machine learning operation), the task arbiter 1002 replaces the task in the fetch queue 1008 with another task descriptor referenced in a different task queue 1004. After the task switch is complete, the task arbiter 1002 may resume the interrupted machine learning operation defined by the task list 904 by storing the replaced task in the fetch queue 1008.

[0096] In some embodiments, the CPU 208 selects a task queue by determining the priority parameter 1110 of the task list 904 (1408). The priority parameter 1110 defines the priority of that task in a machine learning operation in comparison to other tasks in other machine learning operations executed by the neural processor circuit 218. The CPU 208 may facilitate the execution of multiple machine learning operations. Machine learning operations may differ, for example, by using different input data or different neural network architectures and kernel data. The CPU 208 can determine the priority parameter of the task list 904 programmatically or based on user input. The task descriptor 1012 referenced by the task queue 1004 with the highest priority may be selected for execution. The priority of a task queue is determined by the CPU 208 or dynamically determined by information from previously executed task descriptors.

[0097] The task arbiter 1002 determines whether all tasks in task list 904 have been executed (1412). If no tasks in task list 904 have been executed, the process returns to step 1410, where the task arbiter 1002 executes any unexecuted tasks in task list 904.

[0098] If each task in task list 904 is being executed, the task arbiter 1002 removes task list 904 from task queue 1004 (1414). In another embodiment, the reference to the task descriptor 1012 of the task may be removed from task queue 1004 after the task has been executed. The process may then terminate. In another embodiment, the process may return to 1402, where the CPU 208 continues to generate task descriptors for other machine learning operations. In this sense, the process may be repeated to place tasks in the task queue according to a specified task priority for the execution of machine learning operations by the neural processor circuit 218.

[0099] While specific embodiments and uses have been illustrated and described, it should be understood that the present invention is not limited to the exact structures and components disclosed herein, and that various modifications, changes, and variations that would be obvious to those skilled in the art may be made to the configuration, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. A neural engine circuit configured to perform neural actions, A neural task manager circuit and A neural processor circuit comprising, the neural task manager circuit, A fetch queue configured to store the first task descriptor of a task in a first task list in a first task queue, The fetch queue is coupled to the above fetch queue, Obtain the first task descriptor and place the first task descriptor in the fetch queue. In response to determining that the second task queue has a higher priority than the first task queue, the system switches from the task in the first task queue to the task in the second task queue by replacing the first task descriptor stored in the fetch queue with the second task descriptor of the task in the second task list in the second task queue. A task arbiter circuit configured as follows, A configuration queue coupled to the fetch queue and the task arbiter circuit is configured to store configuration data for one or more committed tasks, including committed tasks whose execution has been committed. The task arbiter circuit is further configured to remove the task descriptor or configuration data of the committed task from the configuration queue after the committed task has finished executing. Neural processor circuit.

2. The fetch queue and the task arbiter circuit are further coupled to a task manager direct memory access (DMA), the task arbiter circuit being: The Task Manager DMA is controlled to acquire the first task descriptor or the second task descriptor. The first task descriptor or the second task descriptor is stored in the fetch queue. The neural processor circuit according to claim 1, further configured as follows.

3. The neural processor circuit according to claim 1, wherein the first task queue is coupled to the task arbiter circuit and configured to store a reference to the first task list for instantiating a first neural network, wherein the neural engine circuit is configured to perform the neural operation for the tasks in the first task queue.

4. A kernel direct memory access (DMA) configured to fetch kernel data from system memory located outside the neural task manager circuit and transmit the kernel data to the neural engine circuit in order to execute the neural operation, A buffer DMA configured to receive input data from the system memory and transmit the input data to the neural engine circuit in order to execute the neural operation, The neural processor circuit according to claim 3, further comprising the above.

5. The neural processor circuit according to claim 4, further comprising a data buffer configured to store output data of a task in the first task queue, which is used as input data for a subsequent task in the first task queue in response to the absence of task switching for the task in the first task queue, wherein the data buffer is separate from the system memory located outside the neural task manager circuit.

6. The neural processor circuit according to claim 5, wherein the data buffer is coupled to the buffer DMA and configured to store the input data before transmitting the input data from the system memory to the neural engine circuit in order to perform the neural operation.

7. The neural processor circuit according to claim 1, wherein the first task queue includes a reference to a set of task descriptors, including the first task descriptor, stored in a system memory outside the neural task manager circuit; a first priority parameter indicating a first priority for the first task queue; and a task queue switching parameter indicating whether the first task queue was interrupted by a task switch to the task in the second task queue.

8. The neural processor circuit according to claim 1, wherein the first task descriptor includes configuration data for the task in the first task queue, and the task arbiter circuit is configured to extract configuration data for programming the neural task manager circuit and the neural processor circuit to perform the neural operation.

9. The neural processor circuit according to claim 1, wherein the first task descriptor includes a task switching parameter indicating whether the neural task manager circuit should perform a task switch after the execution of the task in the first task queue.

10. The neural processor circuit according to claim 1, wherein the first task descriptor includes an input surface parameter indicating whether the input data for the task in the first task queue should be obtained from a system memory outside the neural processor circuit, or from a data buffer included in the neural processor circuit and configured to store the output data for the task in the first task queue to be used as input data for subsequent tasks in the first task queue.

11. The neural processor circuit according to claim 1, wherein the first task descriptor includes an output surface parameter indicating whether the output data of the task in the first task queue should be stored in a system memory outside the neural processor circuit or in a data buffer included in the neural processor circuit to facilitate programming of the neural processor circuit.

12. The neural processor circuit according to claim 1, wherein the first task descriptor includes a task switch ready (TSR) parameter that determines whether the neural task manager circuit can perform a task switch after the execution of the task in the first task queue.

13. The neural processor circuit according to claim 1, wherein the first task descriptor includes a source pointer last (SPL) parameter indicating that, after returning to the suspended task queue, the task in the first task queue is the last task with input data stored in system memory outside the neural processor circuit.

14. A method for switching tasks in a neural processor circuit, Obtaining a first task descriptor for a task in a first task list in the first task queue, which includes a memory that stores a first priority parameter that determines the priority of the first task queue, The first task descriptor is stored in the fetch queue of the neural task manager circuit, wherein the first task descriptor is placed in the configuration queue for the execution of the task in the first task queue. The task arbiter circuit of the neural task manager circuit determines that the second task queue has a second priority parameter that has a higher priority than the first priority parameter of the first task queue, The task arbiter circuit performs a task switch from the task in the first task queue to the task in the second task queue by replacing the first task descriptor stored in the fetch queue with the second task descriptor of the task in the second task list in the second task queue. The configuration queue coupled to the fetch queue and the task arbiter circuit stores configuration data for one or more committed tasks, including committed tasks whose execution has been committed. After the committed task has finished executing, the task descriptor or configuration data of the committed task is removed from the configuration queue. A method that includes this.

15. The method according to claim 14, wherein the first task queue includes a reference to a set of task descriptors, including the first task descriptor, stored in a system memory outside the neural task manager circuit, the first priority parameter for the first task queue, and a task queue switching parameter indicating whether the first task queue was interrupted by a task switch to the task in the second task queue.

16. The method according to claim 14, wherein the first task descriptor includes configuration data for the task in the first task queue, and the method further includes extracting configuration data for the task arbiter circuit to program the neural task manager circuit and the neural processor circuit to perform a neural operation.

17. The method according to claim 14, wherein the first task descriptor includes a task switching parameter indicating whether the neural task manager circuit should perform a task switch after the execution of the task in the first task queue.

18. A neural engine circuit configured to perform neural actions, A neural task manager circuit and A system comprising the neural task manager circuit, A fetch queue configured to store first task descriptors for tasks in a first task list in the first task queue, the fetch queue having a memory for storing priority parameters that define the priority of the first task queue, wherein the first task descriptors are located in a configuration queue for the execution of the tasks in the first task queue, and the configuration queue is coupled to the fetch queue and configured to store configuration data for the tasks in the first task queue when the execution of the tasks in the first task queue is committed, The fetch queue and the configuration queue are coupled, Obtain the first task descriptor and place the first task descriptor in the fetch queue. In response to determining that the second task queue has a higher priority than the first task queue, the system switches from the task in the first task queue to the task in the second task queue by replacing the first task descriptor stored in the fetch queue with the second task descriptor of the task in the second task list in the second task queue. A task arbiter circuit configured as follows, The configuration queue is configured to store configuration data for one or more committed tasks, including committed tasks whose execution has been committed, The task arbiter circuit is further configured to remove the task descriptor or configuration data of the committed task from the configuration queue after the committed task has finished executing. system.

19. The fetch queue and the task arbiter circuit are further coupled to a task manager direct memory access (DMA), the task arbiter circuit being: The Task Manager DMA is controlled to obtain the first task descriptor of the task in the first task queue or the second task descriptor of the task in the second task queue. The first task descriptor or the second task descriptor is stored in the fetch queue. The system according to claim 18, further configured as follows.

20. A kernel direct memory access (DMA) configured to fetch kernel data from system memory located outside the neural task manager circuit and transmit the kernel data to the neural engine circuit in order to execute the neural operation, A buffer DMA configured to receive input data from the system memory and transmit the input data to the neural engine circuit in order to execute the neural operation, The system according to claim 18, further comprising the following:

Images