System and method for assigning tasks in neural network processor

By introducing a task manager and neural engine into the neural processor circuit, using task queues and arbitrators to prioritize neural network tasks, the problem of CPU resource-intensive operations is solved, and efficient neural network task management and execution is achieved, reducing the number of neural network tasks. Power and bandwidth consumption.

CN119089954APending Publication Date: 2024-12-06APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411152671.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-05-04
Filing Date
2019-04-04
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art uses a central processing unit (CPU) to perform neural network operations, resulting in computing resource-intensive operations, increasing power consumption and bandwidth consumption, and making it difficult to efficiently manage neural network tasks.

Method used

Neural processor circuits, including neural engines and neural task managers, are adopted to manage neural network tasks through task queues and task arbitrators. Priority parameters are used to retrieve and configure data to realize programming and task execution of neural processors.

Benefits of technology

Effectively managing and executing neural network tasks reduces the burden on the CPU, reduces power consumption and bandwidth consumption, and supports multi-task processing and task switching, improving the efficiency of neural network operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089954A_ABST
    Figure CN119089954A_ABST
Patent Text Reader

Abstract

Embodiments disclosed herein relate to managing tasks to instantiate a neural network when executed by a neural processor circuit. The neural processor circuit includes a neural engine circuit and a neural task manager circuit. The neural task manager circuit includes a plurality of task queues and a task arbiter circuit. Each task queue stores an index of a task list of tasks for machine learning operations. Each task queue may be associated with a priority parameter. Based on the priority of the task queue, the task arbiter circuit retrieves configuration data for tasks from memory external to the neural processor circuit and provides the configuration data to components of the neural processor circuit that include the neural engine circuit. The configuration data programs the neural processor circuit to perform the task. For example, the configuration data may include input data and kernel data that are processed by the neural engine circuitry to perform the task.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with Chinese national application number 201980029309.4, application date April 4, 2019, and invention name “System and method for allocating tasks in a neural network processor”. Technical Field

[0002] The present disclosure relates to circuits for implementing neural networks, and more particularly to managing neural network tasks.

[0003] Related technical description

[0004] An artificial neural network (ANN) is a computing system or model that uses a collection of connected nodes to process input data. ANNs are typically organized into layers, where different layers perform different types of transformations on their inputs. Extensions or variants of ANNs such as convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs) have received a lot of attention. These computing systems or models typically involve a wide range of computational operations, including multiplication and accumulation. For example, CNN is a class of machine learning techniques that primarily uses convolutions between input data and kernel data, which can be decomposed into multiplication and accumulation operations.

[0005] Depending on the type of input data and the operations to be performed, these machine learning systems or models can be configured differently. Such varying configurations will include, for example, preprocessing operations, the number of channels in the input data, the kernel data to be used, the nonlinear functions to be applied to the convolution results, and the application of various post-processing operations. It is relatively easy to instantiate and execute machine learning systems or models of various configurations using a central processing unit (CPU) and its main memory, because such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for various operations of these machine learning systems or models will consume a lot of bandwidth of the central processing unit (CPU) and increase overall power consumption. Summary of the invention

[0006] Embodiments disclosed herein relate to managing tasks that instantiate a neural network when executed by a neural processor circuit. The neural processor circuit includes one or more neural engine circuits and a neural task manager circuit. The neural task manager circuit includes multiple task queue circuits and a task arbiter circuit. Each task queue circuit stores a reference to a task list for tasks for machine learning operations. Each task list and task queue circuit may be associated with a priority parameter. Based on the priority of the task queue circuit, the task arbiter circuit retrieves configuration data for the task from an external memory and provides the configuration data to a component of the neural processor circuit including the one or more neural engines. The configuration data programs the neural processor circuit to perform the task.

[0007] Some embodiments include a method of managing tasks in a neural processor circuit. References to a task list of tasks that instantiate a neural network are stored in a task queue circuit of the neural processor circuit. The task list includes configuration data for the tasks stored in a location in an external memory. The task list is retrieved from the task queue circuit, such as based on a task priority. In response to retrieving the task list, the configuration data is retrieved from the location in the external memory. At least a portion of the configuration data is provided to a neural engine of the neural processor circuit to program the neural engine to perform the tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a high-level diagram of an electronic device according to an embodiment.

[0009] Figure 2 is a block diagram showing components in an electronic device according to one embodiment.

[0010] Figure 3 is a block diagram illustrating a neural processor circuit according to one embodiment.

[0011] Figure 4 is a block diagram of a neural engine in a neural processor circuit according to one embodiment.

[0012] Figure 5 is a conceptual diagram illustrating a circuit for processing input data at a neural processor circuit, according to one embodiment.

[0013] Figure 6 is a conceptual diagram illustrating segmenting input data into slices, tiles, and work units according to one embodiment.

[0014] Figure 7 is a diagram illustrating programming of a rasterizer in a component of a neural processor circuit according to one embodiment.

[0015] Figure 8 is a flow chart illustrating a method of processing input data in a neural processor circuit according to one embodiment.

[0016] Fig. 9 is a schematic block diagram illustrating a neural network represented by a task list according to one embodiment.

[0017] Fig.10 is a block diagram illustrating a neural task manager according to one embodiment.

[0018] Fig.11 is a diagram illustrating the use of a task queue to retrieve a task descriptor according to one embodiment.

[0019] Fig.12 is a diagram illustrating a task descriptor according to one embodiment.

[0020] Fig.13 is a block diagram illustrating an acquisition queue and a configuration queue of a neural task manager according to one embodiment.

[0021] Fig.14 is a flow chart illustrating a method of managing tasks in a neural processor circuit according to one embodiment.

[0022] For purposes of illustration only, the drawings depict and the detailed description describe various non-limiting embodiments. DETAILED DESCRIPTION

[0023] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description to provide a full understanding of the various described embodiments. However, the embodiments may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, thereby not unnecessarily obscuring various aspects of the embodiments.

[0024] Embodiments of the present disclosure relate to managing tasks that instantiate a neural network when executed by a neural processor circuit. Machine learning operations such as inference operations or training operations are defined by a task list of tasks. The neural processor circuit includes one or more neural engines and a neural task manager. The neural task manager includes multiple task queues and a task arbitrator. Each task queue stores a task list of tasks for machine learning operations. Each task list or task queue may be associated with a priority parameter. The task arbitrator retrieves configuration data for the task from an external memory based on the priority parameter, and provides the configuration data to a component of the neural processor circuit including one or more neural engines. In some embodiments, the neural task manager includes a configuration queue that stores configuration data for submitted tasks selected by the task arbitrator, and provides the configuration data to other components of the neural processor circuit. The configuration data programs the neural processor circuit to perform the task. For example, the configuration data may include input data and kernel data processed by the neural engine to perform the task. The configuration data may also include instructions for retrieving and processing the configuration data and instructions for storing output data of the neural engine. Among other things, the neural task manager allows the neural processor circuit to efficiently process multiple machine learning operations. Additionally, the neural task manager may facilitate task switching when higher priority tasks are stored in a task queue while lower priority tasks are being executed.

[0025] A "task" as described herein refers to a processing operation of a neural processor circuit that instantiates a network layer of a neural network, multiple network layers of a neural network, or a portion of a network layer of a neural network. A "task list" as described herein refers to a sequence of tasks, such as a sequence of tasks that instantiate multiple network layers of a neural network, performed by a neural processor circuit.

[0026] Exemplary Electronic Devices

[0027] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a personal digital assistant (PDA) and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Devices, Apple Equipment and Device. Other portable electronic devices, such as wearable devices, laptops, or tablet computers may alternatively be used. In some embodiments, the device is not a portable communication device, but rather a desktop computer or other computing device not designed for portable use. In some embodiments, the disclosed electronic device may include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). Figure 1 The described example electronic device (eg, device 100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0028] Figure 1 is a high-level diagram of an electronic device 100 according to one embodiment. The device 100 may include one or more physical buttons, such as a "home" button or a menu button 104. The menu button 104 is used, for example, to navigate to any application in a set of applications executing on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that recognizes a fingerprint on the menu button 104. The fingerprint sensor can be used to determine whether the finger on the menu button 104 has a fingerprint that matches a fingerprint stored for unlocking the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touch screen.

[0029] In some embodiments, the device 100 includes a touch screen 150, a menu button 104, a push button 106 for turning the device on / off and for locking the device, a volume adjustment button 108, a subscriber identity module (SIM) card slot 110, an earphone jack 112, and a docking / charging external port 124. The push button 106 can be used to turn the device on and off by pressing the button and keeping the button pressed for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 includes various components, including but not limited to a memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), a peripheral device interface, an RF circuit, an audio circuit, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include Figure 1 Components not shown.

[0030] Device 100 is only one example of an electronic device, and device 100 may have more or fewer components than listed above, some of which may be combined into one component or have a different configuration or arrangement. The various components of device 100 listed above are embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing and / or application specific integrated circuits (ASICs).

[0031] Figure 2 1 is a block diagram illustrating components in device 100 according to one embodiment. Device 100 can perform various operations including image processing. For this and other purposes, device 100 may include, among other components, image sensor 202, system on chip (SOC) component 204, system memory 230, persistent storage (e.g., flash memory) 228, motion (orientation) sensor 234, and display 216. Figure 2 The components shown in are merely illustrative. For example, the device 100 may include Figure 2 Other components not shown in the figure (such as speakers or microphones). In addition, some components (such as motion sensor 234) can be omitted from device 100.

[0032] The image sensor 202 is a component for capturing image data and may be implemented as, for example, a complementary metal oxide semiconductor (CMOS) active pixel sensor, a camera, a camcorder, or other device. The image sensor 202 generates raw image data, which is sent to the SOC component 204 for further processing. In some embodiments, the image data processed by the SOC component 204 is displayed on the display 216, stored in the system memory 230, the permanent storage 228, or sent to a remote computing device via a network connection. The raw image data generated by the image sensor 202 may be a Bayer color kernel array (CFA) pattern (hereinafter also referred to as a "Bayer pattern").

[0033] The motion sensor 234 is a component or a group of components for sensing the motion of the device 100. The motion sensor 234 may generate a sensor signal indicating the orientation and / or acceleration of the device 100. The sensor signal is sent to the SOC component 204 for various operations, such as turning on the device 100 or rotating the image displayed on the display 216.

[0034] The display 216 is a component for displaying images generated by the SOC component 204. The display 216 may include, for example, a liquid crystal display (LCD) device or an organic light emitting diode (OLED) device. Based on data received from the SOC component 204, the display 216 may display various images, such as menus, selected operating parameters, images captured by the image sensor 202 and processed by the SOC component 204, and / or other information received from the user interface of the device 100 (not shown).

[0035] System memory 230 is a component for storing instructions executed by SOC component 204 and for storing data processed by SOC component 204. System memory 230 can be embodied as any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, system memory 230 can store pixel data or other image data or statistics in various formats.

[0036] Persistent storage 228 is a component for storing data in a non-volatile manner. Persistent storage 228 retains data even when power is unavailable. Persistent storage 228 may be embodied as a read-only memory (ROM), flash memory, or other non-volatile random access memory device.

[0037] SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. SOC component 204 may include, among other subcomponents, an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and a bus 232 connecting these subcomponents. SOC component 204 may include, among other subcomponents, an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224 Figure 2 More or fewer subcomponents than those shown.

[0038] ISP 206 is hardware that performs various stages of the image processing pipeline. In some embodiments, ISP 206 may receive raw image data from image sensor 202 and process the raw image data into a form usable by other subcomponents of SOC component 204 or components of device 100. ISP 206 may perform various image processing operations, such as image translation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as described below with reference to Figure 3 Described in detail.

[0039] CPU 208 may be implemented using any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. CPU 208 may be a general purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, ARM, or MIPS ISAs, or any other suitable ISA. Figure 2 A single CPU is shown in FIG, but SOC component 204 may include multiple CPUs. In a multi-processor system, each CPU may collectively implement the same ISA, but this is not required.

[0040] Graphics processing unit (GPU) 220 is a graphics processing circuit for performing graphics data. For example, GPU 220 may render objects to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). GPU 220 may include one or more graphics processors that may execute graphics software to perform some or all graphics operations or hardware acceleration of certain graphics operations.

[0041] The neural processor circuit 218 is a circuit that performs various machine learning operations based on calculations including multiplication, addition, and accumulation. Such calculations can be arranged to perform, for example, convolution of input data and kernel data. The neural processor circuit 218 is a configurable circuit that performs these operations in a fast and power-efficient manner while relieving the CPU 208 of resource-intensive operations associated with neural network operations. The neural processor circuit 218 can receive input data from the sensor interface 212, the image signal processor 206, the system memory 230, or other sources such as the network interface 210 or the GPU 220. The output of the neural processor circuit 218 can be provided to various components of the device 100, such as the image signal processor 206, the system memory 230, or the CPU 208 for various operations. The structure and operation of the neural processor circuit 218 are described below with reference to Figure 3 Detailed description.

[0042] The network interface 210 is a subcomponent that supports the exchange of data between the device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data may be received from other devices via the network interface 210 and stored in the system memory 230 for subsequent processing (e.g., via a processor such as described below). Figure 3 The network may include, but is not limited to, a local area network (LAN) (e.g., Ethernet or a corporate network) and a wide area network (WAN). Image data received via network interface 210 may be processed by ISP 206 for image processing.

[0043] The sensor interface 212 is a circuit for interfacing with the motion sensor 234. The sensor interface 212 receives sensor information from the motion sensor 234 and processes the sensor information to determine the orientation or movement of the device 100.

[0044] The display controller 214 is a circuit for sending image data to be displayed on the display 216. The display controller 214 receives image data from the ISP 206, the CPU 208, the graphic processor, or the system memory 230, and processes the image data into a format suitable for display on the display 216.

[0045] Memory controller 222 is a circuit for communicating with system memory 230. Memory controller 222 can read data from system memory 230 for processing by ISP 206, CPU 208, GPU 220, or other subcomponents of SOC component 204. Memory controller 222 can also write data to system memory 230 received from various subcomponents of SOC component 204.

[0046] Video encoder 224 is hardware, software, firmware, or a combination thereof for encoding video data into a format suitable for storage in persistent storage 128 or for passing the data to network interface 210 for transmission over a network to another device.

[0047] In some embodiments, one or more subcomponents of SOC component 204 or some functions of these subcomponents may be performed by software components executing on ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, persistent storage 228, or another device that communicates with device 100 via network interface 210.

[0048] Image data or video data may flow through various data paths within SOC component 204. In one example, raw image data may be generated from image sensor 202 and processed by ISP 206, and then sent to system memory 230 via bus 232 and memory controller 222. After the image data is stored in system memory 230, it may be accessed by video encoder 224 for encoding, or by display 116 for display via bus 232.

[0049] Example Neural Processor Circuit

[0050] The neural processor circuit 218 is a configurable circuit that performs neural network operations on input data based on at least kernel data 340. To this end, the neural processor circuit 218 may include, among other components, a neural task manager 310, a plurality of neural engines 314A to 314N (hereinafter collectively referred to as "plurality of neural engines 314" and also individually referred to as "neural engines 314"), a kernel direct memory access (DMA) 324, a data buffer 318, and a buffer DMA 320. The neural processor circuit 218 may include Figure 3 Other components not shown.

[0051] Each of the neural engines 314 performs computational operations of the neural network operations in parallel. Depending on the operation load, the entire set of neural engines 314 may be operated or only a subset of the neural engines 314 may be operated while the remaining neural engines 314 are placed in a power saving mode to save power. Each of the neural engines 314 includes components for storing one or more kernels, for performing multiply-accumulate operations, and for post-processing to generate output data 328, as described below with reference to Figure 4 Detailed description. An example of a neural network operation is the convolution operation.

[0052] The neural task manager 310 manages the overall operation of the neural processor circuit 218. The neural task manager 310 may receive a list of tasks from a compiler executed by the CPU 208, store the tasks in its task queue, select a task to be executed, and send instructions to other components of the neural processor circuit 218 for executing the selected task. The neural task manager 310 may also perform switching of tasks upon detecting an event, such as receiving an instruction from the CPU 208. In one or more embodiments, the neural task manager 310 sends rasterizer information to components of the neural processor circuit 218 to enable each of these components to track, retrieve, or process appropriate portions of input data and kernel data, as described below with reference to Figures 5 to 7 Although in Figure 3 The neural task manager 310 is shown as part of the neural processor circuit 218 , but the neural task manager 310 may be a component external to the neural processor circuit 218 .

[0053] The kernel DMA 324 is a read circuit that obtains kernel data from a source (e.g., system memory 230) and sends the kernel data 326A-326N to each of the neural engines 314. The kernel data represents information from which kernel elements can be extracted. In one embodiment, the kernel data can be in a compressed format that is decompressed at each of the neural engines 314. Although in some cases the kernel data provided to each of the neural engines 314 can be the same, in most cases the kernel data provided to each of the neural engines 314 is different.

[0054] The data buffer 318 is a temporary storage device for storing data associated with neural network operations. In one embodiment, the data buffer 318 is embodied as a memory accessible by all neural engines 314. The data buffer 318 can store input data 322A to 322N for feeding into the corresponding neural engines 314A to 314N, as well as output from each of the neural engines 314A to 314N for feeding back into the neural engine 314 or sending to a target circuit (e.g., system memory 230). The operation of the data buffer 318 and other components of the neural processor circuit 218 is coordinated so that the input data and intermediate data stored in the data buffer 318 are reused across multiple operations at the neural engine 314, thereby reducing data transfers to and from the system memory 230. The data buffer 318 can operate in a broadcast mode, in which the input data of all input channels are fed to all neural engines 314, or a unicast mode, in which the input data of a subset of the input channels are fed to each neural engine 314.

[0055] The input data 322 stored in the data buffer 318 may be, among other things, image data, a histogram of orientation gradient (HOG) data, audio data, metadata, output data 328 of a previous cycle of the neural engine 314 , and a portion of other processed data received from other components of the SOC component 204 .

[0056] Buffer DMA 320 includes read circuitry that receives a portion (eg, a tile) of input data from a source (eg, system memory 230 ) for storage in data buffer 318 and write circuitry that forwards data from data buffer 318 to a destination (eg, system memory).

[0057] Example Neural Engine Architecture

[0058] Figure 4314 according to one embodiment. The neural engine 314 performs various operations to facilitate neural network operations, such as convolution, spatial pooling, and local response normalization. The neural engine 314 receives input data 322, performs a multiply-accumulate operation (e.g., a convolution operation) on the input data 322 based on the stored kernel data, performs further post-processing operations on the results of the multiply-accumulate operation, and generates output data 328. The input data 322 and / or the output data 328 of the neural engine 314 can be a single channel or multiple channels.

[0059] The neural engine 314 may include, among other components, input buffer circuitry 402, compute core 416, neural engine (NE) control 418, kernel fetch circuitry 432, accumulator 414, and output circuitry 424. The neural engine 314 may include Figure 4 Additional components not shown.

[0060] The input buffer circuit 402 is a circuit that stores a portion of the input data 322 when it is received from the data buffer 318 and sends the appropriate portion 408 of the input data for the current task or processing loop to the computing core 416 for processing. The input buffer circuit 402 includes a shifter 410 that shifts the read position of the input buffer circuit 402 to change the portion 408 of the input data sent to the computing core 416. By changing the portion of the input data provided to the computing core 416 through the shift, the neural engine 314 can perform multiplication and accumulation on different portions of the input data based on a smaller number of read operations. In one or more embodiments, the input data 322 includes data of difference convolution groups and / or input channels.

[0061] The kernel extraction circuit 432 is a circuit that receives the kernel data 326 from the kernel DMA 324 and extracts the kernel coefficients 422. In one embodiment, the kernel extraction circuit 432 references a lookup table (LUT) and uses a mask to reconstruct the kernel from the compressed kernel data 326. The mask indicates the positions in the reconstructed kernel to be padded with zeros and the remaining positions to be padded with numbers. The kernel coefficients 422 of the reconstructed kernel are sent to the compute core 416 to fill the registers in the multiply-add (MAD) circuit of the compute core 416. In other embodiments, the kernel extraction circuit 432 receives the kernel data in an uncompressed format and determines the kernel coefficients without referencing the LUT or using a mask.

[0062] The computation core 416 is a programmable circuit that performs computational operations. To this end, the computation core 416 may include MAD circuits MAD0 to MADN and a post-processor 428. Each of the MAD circuits MAD0 to MADN may store an input value in a portion 408 of the input data and store a corresponding kernel coefficient in a kernel coefficient 422. The input value and the corresponding kernel coefficient are multiplied in each of the MAD circuits to generate a processed value 412.

[0063] The accumulator 414 is a memory circuit that receives and stores processed values ​​412 from the MAD circuit. The processed values ​​stored in the accumulator 414 may be sent back as feedback information 419 for further multiplication and addition operations at the MAD circuit, or sent to the post-processor 428 for post-processing. The accumulator 414 and the MAD circuit combine to form a multiply-accumulator (MAC) 404. In one or more embodiments, the accumulator 414 may have subunits, where each subunit sends data to a different component of the neural engine 314. For example, during a processing cycle, data stored in a first subunit of the accumulator 414 is sent to the MAC circuit while data stored in a second subunit of the accumulator 414 is sent to the post-processor 428.

[0064] The post-processor 428 is a circuit that performs further processing on the value 412 received from the accumulator 414. The post-processor 428 may perform operations including, but not limited to, applying a nonlinear function (e.g., a rectified linear unit (ReLU)), normalized cross-correlation (NCC), merging the results of neural operations performed on 8-bit data into 16-bit data, and local response normalization (LRN). The results of such operations are output from the post-processor 428 to the output circuit 424 as processed values ​​417.

[0065] The NE control 418 controls the operation of other components of the neural engine 314 based on the operating mode and parameters of the neural processor circuit 218. Depending on different operating modes (e.g., group convolution mode or non-group convolution mode) or parameters (e.g., the number of input channels and the number of output channels), the neural engine 314 may operate on different input data in different orders, return different values ​​from the accumulator 414 to the MAC circuit, and perform different types of post-processing operations at the post-processor 428. In order to configure the components of the neural engine 314 to operate in a desired manner, the NE control 418 sends control signals to the components of the neural engine. The NE control 418 may also include a rasterizer 430, which keeps track of the current task or processing loop being processed at the neural engine 314, as described below with reference to FIG. Figures 5 to 7 Detailed description.

[0066] The output circuit 424 receives the processed values ​​417 from the post-processor 428 and interacts with the data buffer 318 to store the processed values ​​417 in the data buffer 318. To this end, the output circuit 424 may emit as output data 328 an order or format different from the order or format in which the processed values ​​417 are processed in the post-processor 428.

[0067] The components in the neural engine 314 may be configured during a configuration cycle by the NE control 418 and the neural task manager 310. To this end, the neural task manager 310 sends configuration information to the neural engine 314 during the configuration cycle. The configurable parameters and modes may include, but are not limited to, the mapping between input data elements and kernel elements, the number of input channels, the number of output channels, the execution of output strides, and enabling / selecting post-processing operations at the post-processor 428.

[0068] Segmenting data for operations processed at a neural processor circuit

[0069] The input data is typically divided into smaller pieces of data for parallel processing at multiple neural engines 314. Multiple cycles of operations are typically performed to generate the output of a task associated with the neural network. The compiler executed by the CPU 208 analyzes the hierarchical structure and nodes of the neural network and determines how to segment the input data based on the hardware constraints of the neural processor circuit 218. One of the functions of the compiler is to determine how to divide the input data into smaller data units for processing at the neural engine 314, and how to iterate the processing in a loop to produce a result for the task.

[0070] Figure 5 is a conceptual diagram illustrating a loop for processing input data at the neural processor circuit 218 according to one embodiment. If a group convolution involving multiple convolution groups is used, the outermost loop represents the processing of the convolution group. A group convolution is a convolution in which the input data of the input channels in each group is used only to generate output data of the output channels of each group but not to generate output data of the output channels of other groups. Therefore, each group of the group convolution can be considered as a separate convolution operation.

[0071] The loop of each convolution group is a processing loop for the input data slice. The entire input data for the convolution operation is segmented into multiple slices in an overlapping manner, such as Figure 6 As shown in FIG. 6 . Overlapping portions 602, 604, 606 are portions of the input data that are over-accessed in two adjacent slices to provide spatial support for the corresponding kernel. The second outermost loop performs a convolution operation for each slice in the input data. Within the loop for the slice is a processing loop for the tiles of the slice. Each slice is segmented into multiple tiles, such as Figure 6As shown. Overlapping portions 608, 610, 612, 614 are portions of the input data in slice 4 that are over-fetched in two adjacent tiles to provide spatial support for the corresponding kernel. The rightmost tile will typically have a width that is smaller than the other tiles of the slice. In one embodiment, the input data for each tile is loaded into data buffer 318 during a read cycle and is reused for operations in the processing loop for the tile. Within the processing loop for the tile is a processing loop for a work unit. Each tile is segmented into multiple work units, such as Figure 6 A work unit is a portion of input data having a size that produces an output value that fits into the accumulator 414 of the neural engine 314 during a single cycle of the compute core 416. Although the shape of each work unit is Figure 6 , but the shape of the work unit may vary depending on the shape and size of the tile. The work unit also has an overlapping portion that represents the fetched data to support the corresponding kernel. In particular, if the tile is tall, the work unit for the last tile of the slice may have the shape of a vertical bar. In one or more embodiments, the size of each work unit is 256 bytes. In such embodiments, for example, the work unit is shaped into one of 16×16, 32×8, 64×4, 128×2, or 256×1 dimensions.

[0072] For each work unit, an internal processing loop may be provided for an output channel group (OCG). The number of output channels generated by a single cycle of the compute core 416 for a given work unit is referred to as an OCG. Depending on the operating mode, each neural engine 314 may process output data for a different number of output channels (e.g., 8 channels, 32 channels) for a single load of input data into its input buffer circuit 402.

[0073] For each output channel group, an inner processing loop may be provided for the input channels (Cin). If input strides are implemented to skip certain input data, a loop for sub-input channels (Sub-Cin) may be provided within the processing loop for the input channels (Cin).

[0074] For each input channel or each sub-input channel, an inner loop is provided for processing the horizontal spatial support of the kernel and the vertical support within each horizontal spatial support. The spatial support refers to the input data for convolution with the kernel and includes over-fetching the input data for performing convolution at the edges of the input data.

[0075] Over-fetching refers to fetching additional input data in the current slice, tile, or work unit so that input data of appropriate dimensions can be provided for convolution with the kernel. In one or more embodiments, over-fetching is performed vertically between slices to obtain additional rows of input data (in Figure 6602, 604, 606), over-access is performed horizontally between tiles to obtain additional columns of input data (in Figure 6 608, 610, 612, 614), and over-acquisition is performed vertically between work units within the tile to obtain additional rows of input data.

[0076] For each spatial support for the kernel, an inner processing loop for an output channel (OC) is provided to generate output data for each output channel (Cout). In the case where the output stride implements spatial upsampling, an additional inner loop is provided for processing each sub-output channel. The loading of kernel coefficients and MAC operations are performed within the loop of the output channel (OC) or sub-output channel (if output stride is implemented) to generate output data for the output channel (OC) or sub-output channel.

[0077] Figure 5 The nested loop structure of is merely illustrative. Depending on various factors, loops may be omitted, added, or structured differently. For example, if only a single convolution group is used, the outermost loop may be removed. In addition, the loop structures for horizontal spatial support and vertical spatial support may be reversed.

[0078] In one or more embodiments, as mentioned above Figure 5 and Figure 6 As described, operations associated with dividing the input space into smaller units and processing these smaller units are performed by rasterizers 714, 718, 720, 722 in various components of the neural processor circuit 218. The rasterizer is a circuit in various components of the neural processor circuit 218 that keeps track of segments (e.g., groups, work units, input channels, output channels) of input / output data and instructs components of the neural processor circuit to correctly process segments of the input data. For example, the rasterizer 720 in the buffer DMA 320 tracks tiles and slices received from the system memory 230, while the rasterizer 718 in the data buffer 318 broadcasts the work units in sequence for processing by the neural engine 314. The rasterizer 722 in the core DMA 322 determines which cores to receive and assigns them to the neural engine 314, while the rasterizer 714 in the neural engine 314 operates the shifter 410 in the input buffer circuit 402 to forward the correct portion 408 of the input data to the MAC 404, and sends the completed output data 328 to the data buffer 318.

[0079] Figure 7is a diagram illustrating programming of rasterizers 714, 718, 720, 722 in components 314, 318, 320, 322 of neural processor circuit 218 according to one embodiment. To perform their functions, each of rasterizers 714, 718, 720, 722 receives task information 710 (e.g., configuration data) indicating how input data and / or kernel data are segmented and processed by each component of neural processor circuit 218. The task information includes information about the details of the current layer (e.g., the dimensions of the input and output data, the dimensions of the associated kernel, the type of padding at the input data boundaries). Rasterizers 714, 718, 720, and 722 may also receive constraints on their operation (e.g., whether tile widths are allowed or not allowed to exceed a threshold).

[0080] By providing rasterizers in different components of the neural processor circuit 218, the overhead of data transmitted between components of the neural processor circuit 218 may be reduced. If a single central rasterizer is provided to control different components of the neural processor circuit 218, kernel data, input data, and output data transmitted between these components may need to identify associated locations in the task loop, such as convolution groups, tiles, slices, work units, input channels, and output channels. By using a distributed rasterizer, separate metadata is not required to transmit kernel data, input data, and output data between components of the neural processor circuit 218.

[0081] Example process at the Neural Engine architecture

[0082] Figure 8 is a flow chart illustrating a method of processing input data in the neural processor circuit 218 according to one embodiment. The method may include different and / or additional steps, or the steps may be in a different order.

[0083] After the neural task manager 310 programs the rasterizers 714, 718, 720, 722, the process of operating the buffer DMA 320 is initiated by the rasterizer 720 instructing 804 the buffer DMA 320 to cause the buffer DMA 320 to receive tiles of input data from the system memory 230. The tiles received by the buffer DMA 320 are stored 806 in the data buffer 318.

[0084] The rasterizer 718 in the data buffer 318 then instructs 808 the data buffer 318 to send the work unit to the one or more neural engines 314. The work unit is then stored in the input buffer circuitry 402 of the one or more neural engines 314.

[0085] In one or more embodiments, the input buffer circuit 402 selects 816 a portion of the work unit to be sent to the MAC 404 to perform a multiply-accumulate operation. The MAC 404 then performs 820 a multiply-accumulate operation on the selected portion of the work unit using the corresponding core. It is then determined 824 whether the entire work unit is processed at one or more neural engines 314. If not, the selected portion of the work unit is shifted 828 by the shifter 410 and returns to perform 820 another round of multiply-accumulate operations.

[0086] If it is determined 824 that the entire work unit is processed, it continues to determine 832 whether all work units in the tile are processed. If not, the process advances 836 to the next work unit by causing the data buffer 318 to send 808 the next work unit to one or more neural engines 314, and repeats the subsequent process.

[0087] If it is determined 832 that all work units in the tile are processed by the neural engine 314, the process continues to determine 840 whether all tiles for input data are processed. If not, the process advances 844 to the next tile by causing the rasterizer 720 to instruct 804 the buffer DMA 320 to receive the next tile from the system memory 230 and repeating the subsequent process.

[0088] If it is determined 840 that all tiles of the input data are processed, the process ends for the current input data. Then, the process can be repeated to process the next input data or proceed to the next task.

[0089] References Figure 8 The embodiments of the process described are merely illustrative. Additional circuits may be implemented, as described above with reference to Figure 5 Furthermore, the order of the processes may be modified or omitted.

[0090] Example Neural Task Manager Architecture

[0091] A neural network may include network layers or sub-layers that are instantiated or implemented as a series of tasks performed by the neural processor circuit 218 . Fig. 9 9 is a schematic block diagram showing a neural network 900 represented by a task list 904 according to one embodiment. The neural network 900 includes a network layer (or sublayer), which includes convolutional layers C1, C2, C3 (including sublayer C3 00 、C3 10 、C3 11 、C3 20 and C3 21), C5 and C6, and pooling layers P2 and P4. Neural network 900 is an example of a neural network architecture that may be instantiated by neural processor circuit 218. That is, when neural network 900 is converted into task list 904 to become executable by neural processor circuit 218. Other types of neural network architectures having different types of network layers or network layer orders may also be instantiated by neural processor circuit 218.

[0092] Neural network 900 is converted into task list 904, such as by CPU 208. Task list 904 comprises a linear linked list defining a series of tasks, the series of tasks comprising task C1, task C2+P2, task C3+P4, task C4+P5, task C5+P6, task C6+P7, task C7+P8, task C8+P9, task C9+P10, task C11+P12, task C12+P13, task C14+P14, task C15+P15, task C16+P16+P17, task C17+P18, task C19+P19, task C20+P19, task C30+P19, task C40+P19, task C5 00 +P4, Mission C3 10 、Task C3 11 +P4, Mission C3 20 、Task C3 21 +P4, Mission C5 a 、Task C5 b and task C6. Each task is associated with a task descriptor that defines the configuration of the neural processor circuit 218 to perform the task. Each task may correspond to a single network layer of the neural network 900, a portion of a network layer of the neural network 900, or multiple network layers of the neural network 900. For example, task C1 corresponds to a single network layer C1, task C2+P2 corresponds to multiple network layers C2 and P2, and task C5 a and C5 b Each corresponds to a portion of network layer C5. Neural processor circuit 218 instantiates neural network 900 by executing tasks of task list 904 under the control of neural task manager 310.

[0093] Fig.10 is a block diagram illustrating a neural task manager 310 according to one embodiment. The neural task manager 310 manages tasks performed by the neural processor circuit 218 for one or more neural networks 900. The neural task manager 310 may include, among other components, a task arbitrator 1002, task queues 1004A through 1004N (hereinafter collectively referred to as "task queues 1004" and also individually as "task queues 1004"), a task manager direct memory access (DMA) 1006, a fetch queue 1008, and a configuration queue 1010. The neural task manager 310 may include Fig.10 Other components not shown.

[0094] The task arbiter 1002 is a circuit or a combination of circuits and firmware that selects tasks from the task queue 1004 for execution by the neuro-processor circuit 218. The task arbiter 1002 dequeues the task from the task queue 1004 and places the task in the configuration queue 1010. While the task is in the configuration queue, it is submitted for execution, and the neuro-processor circuit performs pre-fetching of input data and kernel data before the task is executed by other components of the neuro-processor circuit 218. For example, the task arbiter 1002 may perform fixed priority arbitration between multiple task queues 1004 and select a task from the task queue 1004 with the highest priority for retrieval of the task descriptor 1012 from the system memory 230 by the task manager DMA 1006.

[0095] The neural task manager 310 may include one or more task queues 1004. Each task queue 1004 is coupled to the CPU 208 and the task arbiter 1002. Each task queue 1004 receives references to the task list 904 of tasks from the CPU 208 that, when executed by the neural processor circuit 218, instantiate the neural network 900. The references stored in each task queue 1004 may include a set of pointers and counters to the task list 904 of the task descriptor 1012 in the system memory 230. Each task queue 1004 may also be associated with a priority parameter that defines the relative priority of the task queue 1004. The task descriptor 1012 of the task specifies the configuration of the neural processor circuit 218 for executing the task.

[0096] The task manager DMA 1006 is coupled to the task arbiter 1002, the system memory 230, and the fetch queue 1008. The task manager DMA 1006 includes a read circuit that receives a task descriptor 1012 of a task from a source (e.g., the system memory 230) to store in the fetch queue 1008. For example, the task arbiter 1002 selects a task queue 1004 according to the priority of the task queue 1004, and uses the task list 904 indexed by the selected task queue 1004 to control the task manager DMA 1006 to select the task descriptor 1012 of the task.

[0097] The acquisition queue 1008 is a single entry queue that stores task descriptors 1012 of tasks that are pending submission for execution. The acquisition queue 1008 is coupled to the task manager DMA 1006 to receive the task descriptors 1012 from the system memory 230 and provide the task descriptors 1012 to the configuration queue 1010, or provide the configuration data 1014 extracted from the task descriptors 1012 to the configuration queue 1010.

[0098] The configuration queue 1010 holds configuration data 1014 for a plurality of tasks that have been submitted for execution. While a task is in the configuration queue 1010, the kernel DMA 324 may fetch kernel data from the system memory 230 for storage in the kernel fetch circuit 432 of the neural engine 314, and the buffer DMA 320 may fetch input data from the system memory 230 for storage in the data buffer 318. To execute the task, the kernel fetch circuit 432 provides the pre-fetched kernel data to the MAC 404 of the neural engine 314, and the data buffer 318 provides the pre-fetched input data to the MAC 404 of the neural engine 314. In some embodiments, the configuration queue 1010 may include a plurality of queues that hold configuration data 1014 fetched from submitted task descriptors 1012. As shown in conjunction with Fig.13 Discussed in more detail, configuration queue 1010 is further coupled to other components of neural processor circuit 218 to configure neural processor circuit 218 according to configuration data 1014 .

[0099] Fig.11 1 is a diagram illustrating the use of a task queue 1004 to retrieve a task descriptor 1012 according to one embodiment. The task queue 1004 includes references, such as a set of pointers, to task descriptors 1012A to 1012N stored in the system memory 230. To this end, the task queue 1004 may include memory storing a header parameter 1102, a network identifier (ID) 1104, a base index 1106, a tail parameter 1108, a count parameter 1110, and a priority parameter 1112. The header parameter 1102 is a pointer to the location of the system memory 230 where the task descriptor 1012A is stored at the head of the task queue 1004. The network ID 1104 identifies the neural network 900 of the task descriptor 1012 at the head of the task queue 1004, and the base index 1106 is an index into the base table 1114 of the network ID 1104 marked with the task descriptor 1012A at the head of the task queue 1004. The count parameter 1110 defines the number of task descriptors 1012 in the task queue 1004. The priority parameter 1112 defines the priority of the task queue 1004, which the task arbiter 1002 uses to select between the plurality of task queues 1004.

[0100] When a particular task queue 1004 is selected (e.g., based on the priority parameter 1112), the task arbiter 1002 indexes the header parameter 1102, the network ID 1104, the base address index 1106, and the base address table 1114 for retrieving the task descriptor 1012 from the system memory 230, and places the task descriptor 1012 into the acquisition queue 1008 to initiate a commitment for execution of the task. In each configuration cycle, the task arbiter 1002 may continue to place the task descriptor 1012 into the acquisition queue 1008 according to the order of the tasks defined by the task list 904 of the task queue 1004 (such as by retrieving the next task descriptor 1012B, etc.).

[0101] Fig.12 1008 and 1010. The task descriptor 1012 is a diagram illustrating a task descriptor 1012 according to one embodiment. The task arbiter 1002 places the task descriptor 1012 from the system memory 230 in the acquisition queue 1008 and then transfers it to the configuration queue 1010. The highest priority (e.g., advanced) task descriptor 1012 in the configuration queue 1010 is used to configure the neural processor circuit 218 for execution during the configuration cycle. The task descriptor 1012 includes configuration data 1014, which includes a task descriptor header 1202 and address data 1204A to 1204N (hereinafter referred to as "address data 1204"). The task descriptor header 1202 includes configuration data 1014 that configures various operations of the neural task manager 310, including operations related to task selection and task switching. For example, the task descriptor header 1202 can be parsed by the task arbiter 1002 to extract the configuration data 1014 that programs the neural task manager 310 and other components of the neural processing circuit 218. The task descriptor header 1202 may include a task identifier (ID) 1206 identifying the task, a neural network identifier (ID) 1208 identifying the neural network 900 instantiated by the task, a task switch parameter 1210 defining whether the neural task manager 310 should initiate a task switch after executing the task (e.g., execute a task of a different task queue 1004), an input surface parameter 1212 defining whether input data for the task should be retrieved from the system memory 230 or the data buffer 318, an output surface parameter 1214 defining whether output data for the task should be stored in the system memory 230 or the data buffer 318, various (e.g., base address) pointers 1216 facilitating programming of the neural processor circuit 218, and one or more debug / exception parameters 1218 that control events, exceptions, or debug logging.

[0102] Each instance of address data 1204A to 1204N (collectively or individually referred to as "address data 1204") defines an address and data payload pair for programming components of neural processor circuit 218. The data payload may include input data and kernel data for performing a task. For example, each instance of address data 1204 includes register data defining the data payload, a register address defining the destination memory location of neural processing circuit 218 for receiving the register data, and a register count defining the number of consecutive memory locations (e.g., registers) to which the pending register data is written. In some embodiments, the register address is combined with a base address stored in header 1202 to define the complete address of each memory location. If task descriptors 1116 are generated at compile time, the actual run-time addresses may be unknown. Use of base address table 1114 avoids copying or updating all task descriptors with dynamically assigned addresses.

[0103] Fig.131 is a block diagram illustrating an acquisition queue 1008 and a configuration queue 1010 according to one embodiment. The configuration queue 1010 is coupled to the acquisition queue 1008, which is coupled to the system memory 230 via the task manager DMA 1006. The configuration queue 1010 is also coupled to the rasterizer 714 of one or more neural engines 314, the rasterizer 718 of the data buffer 318, the rasterizer 720 of the buffer DMA 320, and the rasterizer 722 of the kernel DMA 322. The acquisition queue 1008 stores task descriptors 1012 (e.g., including task descriptor headers 1202 and address data 1204A to 1204N) for tasks that are pending and not submitted for execution. The acquisition queue 1008 reduces the latency of reading the next task descriptor 1012 from the system memory 230 into the configuration queue 1010. The acquisition queue 1008 stores the highest priority task descriptor 1012 determined by the task arbiter 1002. If a higher priority task descriptor 1012 has been enqueued (e.g., enqueued from a higher priority task queue 1004), the task arbiter 1002 may replace the task descriptor 1012 stored in the acquisition queue 1008. The task descriptor 1012 in the acquisition queue 1008 does not initiate input data or kernel prefetching and does not affect the task queue priority, pointers, or counters. In this way, by writing the higher priority task descriptor 1012 to the acquisition queue 1008, the task descriptor 1012 in the acquisition queue 1008 can be easily replaced by the higher priority task descriptor 1012. When the task descriptor 1012 stored in the configuration queue 1010 is executed by the neural processor circuit 218, the task descriptor 1012 stored in the acquisition queue 1008 is transferred to the configuration queue 1010, and another task descriptor 1012 of the subsequent task can be stored in the acquisition queue 1008.

[0104] The configuration queue 1010 stores task descriptors 1012 submitted for tasks to be performed by the neural processor circuit 218. In some embodiments, the configuration queue 1010 includes a plurality of separate queues 1310, each of which stores a portion of the configuration data 1014 (including configuration data 1014A to 1014E) extracted from the task descriptors 1012. In addition, the queues 1310 are each coupled to a corresponding component of the neural processor circuit 218 for programming the component with the configuration data 1014. Through the operation of the configuration queues 1010, the neural task manager 310 programs the rasterizers 714, 718, 720, 722 to perform the tasks described above in Figure 7 and Figure 8For example, queue 1310A is coupled to rasterizer 714 of neural engine 314 to provide configuration data 1014A that controls the operation of shifter 410 in input buffer circuit 402 to forward correct portion 408 of input data to MAC 404 and send completed output data 328 to data buffer 318. Queue 1310B is coupled to rasterizer 718 of data buffer 318 to provide configuration data 1014B that controls data buffer 318 to broadcast input data (e.g., work units) for processing by neural engine 314. Queue 1310C is a read queue that is coupled to rasterizer 720 of buffer DMA 320 to provide configuration data 1014C that controls buffer DMA 320 to retrieve input data (e.g., tiles) from system memory 230 and store the input data in data buffer 318. Queue 1310D is a write queue that is coupled to the rasterizer 720 of the buffer DMA 320 to provide configuration data 1014D that controls the buffer DMA 320 to store output data in the system memory 230. Queue 1310E is coupled to the rasterizer 722 of the core DMA 322 to provide configuration data 1014E that controls which cores are to receive and be assigned to the neural engine 314. In some embodiments, the task descriptors 1012 or configuration data 1014 stored in the configuration queue 1010 cannot be replaced and will be executed in a first-in, first-out (FIFO) order.

[0105] Example process at the Neural Task Manager architecture

[0106] Fig.14 is a flow chart illustrating a method of managing tasks in the neural processor circuit 218 according to one embodiment. The method may include different and / or additional steps, or the steps may be in a different order.

[0107] The CPU 208 generates 1402 a task list 904 of task descriptors 1012 of tasks that, when executed by the neural processor circuit 218, instantiate the neural network 900. For example, the CPU 208 may receive input data for a machine learning operation from the image sensor 202, the system memory 230, the permanent storage device 228, the network interface 210, or some other component. The machine learning operation may include an inference operation or a training operation. The neural network 900 may include kernel data and a neural network architecture including network layers. The input data is applied to the neural network 900 to perform the machine learning operation. The kernel data and network layers of the neural network 900 may be calculated during the machine learning training process. The CPU 208 performs a compilation operation (offline or on the fly) to transform the neural network description 900 into a linked list of task descriptors 1012, referred to herein as a task list 904. Each task is defined by a task descriptor 1012, and when executed by the neural processor circuit 218, each task instantiates a single network layer, multiple network layers, or a portion of a network layer. Each task descriptor 1012 for a task includes configuration data 1014, such as a task descriptor header 1202 and address data 1204 defining a pair of addresses and data payloads for the neural processor circuit 218. The data payload may include kernel data and input data for the neural network 900. The configuration data 1014 also includes instructions for configuring the operation of the rasterizers 714, 718, 720, and 722 to perform the task.

[0108] The CPU 208 stores 1404 the task list 904 of the task descriptor 1012 in the system memory 230. In some embodiments, the CPU 208 or another CPU external to the electronic device 100 generates the task descriptor 1012 and stores the task descriptor 1012 in the permanent storage device 228 or some other non-volatile memory. The task descriptor 1012 is loaded into the system memory 230 along with the kernel data and the input data for use by the neural processor circuit 218. The CPU 208 can be coupled to the system memory 230 via the bus 232 and the memory controller 222.

[0109] The task list 904 may include a set of pointers that reference the location of the task descriptor 1012 in the system memory 230. In addition, the CPU 208 may update parameters of the task descriptor 1102, such as a storage address or a network ID. For example, the task list 904 may include a header parameter 1102 and a trailer parameter 1108 that define the starting register address and the ending register address of the system memory 230, respectively, in which the plurality of task descriptors 1012 are stored. In some embodiments, the references to the register addresses in the task list 904 may be partial addresses, and the base address table 1114 is used to define the complete referenced address to the system memory 230. In some embodiments, the CPU 208 may patch the absolute address as needed. The CPU 208 may further configure the neuroprocessor circuit 218 by setting its base address register.

[0110] The task arbiter 1002 of the neural task manager 310 enqueues 1406 the task list 904 for execution by the neural processor circuit 218. For example, the neural task manager 310 includes a plurality of task queues 1004A to 1004N. The task queues 1004 may each store a reference to a task list 904. In addition, the task queues 1004 are prioritized for execution according to the priority parameters 1112 of the task lists 904 referenced by the task queues 1004.

[0111] In some embodiments, the CPU 208 performs general configuration of the neural processor circuit 218 to perform the task. The CPU 208 may further start the neural processor circuit 218 if it is not already running.

[0112] The task arbiter 1002 selects 1408 the task queue with the highest priority for execution. For example, in each programmed time period, the task arbiter 1002 selects a task queue 1004 based on a priority parameter of the task queue 1004 or a comparison of the task list 904 of the task queue 1004, and executes the task from the task list 904 from the highest priority task queue 1004.

[0113] The neural processor circuit 218 executes 1410 the task from the selected task queue. For example, the neural processor circuit 218 executes Figure 8 The configuration queue 1010 of the neural task manager 310 may provide configuration data 1014 (or task information 710) to the rasterizers 714, 718, 720, 722 of the neural processor circuit 218 to program the neural processor circuit 218 to perform the task, such as Figure 7In addition, the execution of the task may include processing pre-fetched kernel data and input data while the task is in the configuration queue 1010. The execution of each task may include multiple processing loops for processing input data, such as Figure 5 and Figure 6 The task arbitrator 1002 may dequeue the task from the configuration queue 1010 after executing the task.

[0114] In some embodiments, to execute a task, the task arbiter 1002 controls the task manager DMA 1006 to retrieve the task descriptor 1012 of the task of the task list 904 and store the task descriptor 1012 in the fetch queue 1008. After executing a previously submitted task, the task arbiter 1002 may dequeue the executed task by removing the task descriptor or the configuration data of the task from the configuration queue 1010. The task descriptor 1012 of the current task or the fetched configuration data 1014 is then placed in the configuration queue 1010 from the fetch queue 1008. When the task is in the configuration queue 1010, the neuro-processor circuit 218 may initiate a pre-fetch operation of the kernel DMA 324 on the kernel data from the system memory 230 to the kernel fetch circuit 432, and a pre-fetch operation of the buffer DMA 320 on the input data from the system memory 230 to the data buffer 318.

[0115] If the different task queue has a higher priority, a task switch to the task queue with the higher priority may be performed if possible. To perform a task switch (e.g., to another machine learning operation), the task arbitrator 1002 replaces the task in the acquisition queue 1008 with another task descriptor referenced in a different task queue 1004. The task arbitrator 1002 may resume the interrupted machine learning operation defined by the task list 904 after the task switch is completed by storing the replaced task in the acquisition queue 1008.

[0116] In some embodiments, the CPU 208 determines 1408 a priority parameter 1110 of the task list 904 to select a task queue. The priority parameter 1110 defines the priority of the task of the machine learning operation relative to other tasks of other machine learning operations performed by the neural processor circuit 218. The CPU 208 can facilitate the execution of multiple machine learning operations. The machine learning operations can be different, such as by using different input data or different neural network architectures and kernel data. The CPU 208 can determine the priority parameter of the task list 904 programmatically or based on user input. The task descriptor 1012 indexed by the task queue 1004 with the highest priority can be selected for execution. The priority of the task queue is determined by the CPU 208, or dynamically determined by information from previously executed task descriptors.

[0117] Task arbitrator 1002 determines 1412 whether all tasks of task list 904 have been executed. If the tasks of task list 904 have not been executed, the process returns to step 1410 where task arbitrator 1002 executes the unexecuted tasks of task list 904.

[0118] If each task in the task list 904 has been executed, the task arbiter 1002 removes 1414 the task list 904 from the task queue 1004. For another example, after executing the task, the reference to the task descriptor 1012 of the task can be removed from the task queue 1004. The process can end. For another example, the process can return to 1402, where the CPU 208 can continue to generate task descriptors for tasks for other machine learning operations. In this sense, the process can be repeated to place tasks into the task queue according to the specified task priority for execution of machine learning operations by the neuro-processor circuit 218.

[0119] While specific embodiments and applications have been illustrated and described, it should be understood that the invention is not limited to the precise construction and components disclosed herein, and that various modifications, changes and variations that will be apparent to those skilled in the art may be made to the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope of the present disclosure.

Claims

1. A neural processor circuit, comprising: Neural Engine Circuitry; as well as A neural task manager circuit coupled to the neural engine circuit, the neural task manager circuit comprising: The first part of the direct memory access DMA device; an acquisition queue coupled to the first portion of the DMA device; a configuration queue, the configuration queue being coupled to the acquisition queue; a task queue circuit configured to store references to a task list of task descriptors, the task list comprising tasks for instantiating a neural network, the task descriptors of the task list comprising configuration data for the tasks; and a task arbiter circuit coupled to the task queue circuit, the first portion of the DMA device, and the fetch queue, wherein the task arbiter circuit is configured to: retrieving the task descriptor stored in a memory external to the neural processor circuit based on the reference to the task list stored in the task queue circuit; storing the task descriptor received from the memory in the first portion of the DMA device; providing the task descriptors from the first portion of the DMA device to the fetch queue, wherein the fetch queue is configured to store the task descriptors for the tasks that are pending and not submitted for execution; transferring the task descriptor from the acquisition queue to the configuration queue to become a dequeued task submitted for execution; and A portion of the configuration data for the dequeued task is provided to the neural engine circuitry, wherein the neural engine circuitry is configured to perform the dequeued task based on the portion of the configuration data.

2. The neural processor circuit of claim 1 , wherein the dequeued task instantiates a single network layer of the neural network, multiple network layers of the neural network, or a portion of a network layer of the neural network.

3. A neural processor circuit according to claim 1, wherein the task arbitrator circuit is further configured to store the configuration data included in the task descriptor in the configuration queue of the neural task manager circuit, the configuration queue is coupled to the neural engine circuit and is configured to provide the portion of the configuration data to the neural engine circuit.

4. The neural processor circuit of claim 1 , further comprising: a second portion of the DMA device configured to retrieve kernel data for the dequeued task from the memory external to the neural processor circuit in response to the configuration data being stored in the configuration queue; and A third portion of the DMA device is configured to retrieve input data for the dequeued task from the memory external to the neuro-processor circuit in response to the configuration data being stored in the configuration queue.

5. The neural processor circuit of claim 1 , wherein the task queue circuit is a first task queue circuit, and wherein the neural task manager circuit further comprises: a second task queue circuit coupled to the task arbiter circuit, wherein the first task queue circuit has a first priority and the second task queue circuit has a second priority; and The task arbiter circuit is further configured to: store one of the configuration data of the dequeued task or another configuration data of another task in the acquisition queue based on a comparison of the first priority and the second priority.

6. The neural processor circuit of claim 1 , further comprising a data buffer coupled to the memory and the neural engine circuit external to the neural processor circuit, wherein the task arbitrator circuit is further configured to provide another portion of the configuration data to the data buffer, the other portion of the configuration data programming the data buffer to broadcast a work unit of input data of the dequeued task to the neural engine circuit.

7. The neural processor circuit of claim 6, further comprising a fourth portion of the DMA device, the fourth portion of the DMA device coupled to the data buffer and the memory external to the neural processor circuit, and wherein: The task arbiter circuit is further configured to provide a third portion of the configuration data to the fourth portion of the DMA device, the third portion of the configuration data being configured to: program the fourth portion of the DMA device to retrieve a tile of the input data for the dequeued task from the memory external to the neural processor circuit and store the tile in the data buffer; as well as The tile includes a plurality of work units.

8. The neural processor circuit of claim 1 , wherein: The neural engine circuit comprises: an input buffer circuit coupled to the neural task manager circuit; and a multiply-add MAD circuit, the multiply-add MAD circuit being coupled to the input buffer circuit; The portion of the configuration data is configured to program the input buffer circuit to provide a portion of the input data of the dequeue task stored in the input buffer circuit to the MAD circuit.

9. The neural processor circuit of claim 8, further comprising a data buffer coupled to the memory and the neural engine circuit external to the neural processor circuit, wherein the neural engine circuit further comprises an output circuit, and wherein the portion of the configuration data is configured to program the output circuit to provide output data from the MAD circuit to the data buffer.

10. The neural processor circuit of claim 1 , further comprising a fifth portion of the DMA device, the fifth portion of the DMA device coupled to the memory and the neural engine circuit external to the neural processor circuit, and wherein: The task arbiter circuit is further configured to provide another portion of the configuration data to the fifth portion of the DMA device, and the other portion of the configuration data is configured to: program the fifth portion of the DMA device to retrieve kernel data from the memory external to the neural processor circuit and provide the kernel data to the neural engine circuit to perform the dequeue task.

11. A method for managing tasks in a neural processor circuit, comprising: storing, in a task queue circuit of the neural processor circuit, a reference to a task list of task descriptors, the task list including tasks for instantiating a neural network, the task descriptors in the task list including configuration data for the tasks; at a task arbiter circuit coupled to the task queue, retrieving the task descriptor stored in a memory external to the neuro-processor circuit based on the reference to the task list stored in the task queue circuit; storing the descriptor of the task received from the memory in a direct memory access (DMA) device of the neural processor circuit; providing the task descriptor for the task from the DMA device to a fetch queue, the fetch queue being configured to store the task descriptors for the tasks that are pending and not submitted for execution; transferring the task descriptor from the acquisition queue to a configuration queue to become a dequeued task submitted for execution, wherein the configuration queue is configured to store the dequeued task; as well as A portion of the configuration data for the dequeued task is provided to neural engine circuitry, the neural engine circuitry being configured to perform the dequeued task based on the portion of the configuration data.

12. The method of claim 11, wherein the dequeued task instantiates a single network layer of the neural network, a plurality of network layers of the neural network, or a portion of a network layer of the neural network.

13. The method according to claim 11, further comprising: storing the configuration data in the configuration queue of the neural processor circuit; as well as providing the portion of the configuration data from the configuration queue to the neural engine circuitry; When the configuration data is stored in the configuration queue, retrieving the kernel data of the dequeued task from the external memory; and When the configuration data is stored in the configuration queue, input data of the dequeued task is retrieved from the external memory.

14. The method according to claim 13, further comprising: retrieving the configuration data for the dequeued task from the memory via the DMA device of the neural processor circuit; as well as storing the configuration data in the acquisition queue of the neural processor circuit; and When another configuration data of the executed task is removed from the configuration queue, the configuration data is provided from the acquisition queue to the configuration queue.

15. The method of claim 14, wherein the task queue circuit is a first task queue circuit, the neural processor circuit further comprises a second task queue circuit, and the method further comprises: storing a first priority corresponding to the first task queue circuit and a second priority corresponding to the second task queue circuit; as well as Based on a comparison of the first priority and the second priority, one of the configuration data of the dequeued task or another configuration data of another task is stored in the get queue.

16. The method of claim 11 further comprising providing another portion of the configuration data to a data buffer of the neural processor circuit, the another portion of the configuration data programming the data buffer to broadcast a work unit of input data of the dequeued task to the neural engine circuit.

17. A system comprising: a memory device configured to store a task descriptor; as well as a neural processor circuit coupled to the memory device and comprising: Neural Engine circuitry; and A neural task manager circuit coupled to the neural engine and comprising: Direct memory access DMA device; an acquisition queue coupled to the DMA device; a configuration queue, the configuration queue being coupled to the acquisition queue; a plurality of task queue circuits, wherein a task queue circuit of the plurality of task queue circuits is configured to store references to task lists of task descriptors, the task lists comprising tasks for instantiating a neural network, and wherein the task descriptors of the task lists comprise configuration data for the tasks; and a task arbiter circuit, the task arbiter circuit being coupled to the plurality of task queue circuits, the DMA device, and the acquisition queue, the task arbiter circuit being configured to: storing the task descriptor from the memory into the DMA device, providing the task descriptors from the DMA device to the fetch queue, wherein the fetch queue is configured to store the task descriptors for the tasks that are pending and not submitted for execution, transferring the task descriptor from the acquisition queue to a configuration queue to become a dequeued task submitted for execution, and A portion of the configuration data for the dequeued task is provided to the neural engine circuitry, wherein the neural engine circuitry is configured to perform the dequeued task based on the portion of the configuration data.

18. The system of claim 17, wherein the dequeued task instantiates a single network layer of the neural network, a plurality of network layers of the neural network, or a portion of a network layer of the neural network.

19. A system according to claim 17, wherein the task arbitrator circuit is further configured to store the configuration data included in the task descriptor in the configuration queue of the neural task manager circuit, the configuration queue is coupled to the neural engine circuit and is configured to provide the portion of the configuration data to the neural engine circuit.

20. The system of claim 17, wherein: The task queue circuit is a first task queue circuit, and wherein the plurality of task queue circuits further comprises a second task queue circuit coupled to the task arbiter circuit, the first task queue circuit having a first priority and the second task queue circuit having a second priority; and The task arbiter circuit is further configured to: store one of the configuration data of the dequeued task or another configuration data of another task in the acquisition queue based on a comparison of the first priority and the second priority.