Branch operations for neural processor circuitry

Abstract

A neural processor includes a neural engine to perform a convolution operation on input data corresponding to one or more tasks to generate output data. The neural processor circuit also includes a data processor circuit coupled to the one or more neural engines. The data processor circuit receives the output data from the neural engine and generates a branch command from the output data. The neural processor circuit also includes a task manager coupled to the data processor circuit. The task manager receives the branch command from the data processor circuit. The task manager enqueues one of two or more segmented branches according to the received branch command. The two or more segmented branches are after a pre-branch task segment that includes a pre-branch task. The task manager transmits a task from a selected one of the segmented branches to a data processor circuit to execute the task.

Description

Background Technology 1. Technical Field

[0002] This disclosure relates to a circuit for performing operations related to a neural network, and more specifically to operations related to branching and selecting task segments in a neural processor circuit.

[0003] 2. Relevant Technical Descriptions

[0004] Artificial neural networks (ANNs) are computational systems or models that use a set of connected nodes to process input data. ANNs are typically organized into layers, with different layers performing different types of transformations on their inputs. Extensions or variants of ANNs such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs) have received considerable attention. These computational systems or models typically involve a wide range of computational operations, including multiplication and accumulation. For example, CNNs are a class of machine learning techniques that primarily use convolutions between input data and kernel data; convolutions can be decomposed into multiplication and accumulation operations.

[0005] These machine learning systems or models can be configured differently depending on the type of input data and the operations to be performed. Such varied configurations would include, for example, preprocessing operations, the number of channels in the input data, the kernel data to be used, the nonlinear functions to be applied to the convolution results, and the application of various post-processing operations. Instantiating and executing machine learning systems or models with various configurations using a central processing unit (CPU) and its main memory is relatively straightforward, as such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for the various operations of these machine learning systems or models would consume significant CPU bandwidth and increase overall power consumption.

[0006] Electronic devices equipped with neural processors dedicated to performing computations related to machine learning models have become increasingly common. Due to the growing reliance on artificial intelligence in various software applications, electronic devices often operate multiple software applications that run one or more neural networks. Summary of the Invention

[0007] The embodiment relates to a neural processor circuit including one or more neural engine circuits for performing convolution operations on input data corresponding to a pre-branching task to generate output data. The neural processor circuit also includes a data processor circuit coupled to the one or more neural engine circuits. The data processor circuit receives the output data from the neural engine circuits and generates branch commands from the output data. The neural processor circuit also includes a task manager circuit coupled to the data processor circuit. The task manager circuit receives the branch commands from the data processor circuit. The task manager circuit enqueues one of two or more segmented branches according to the received branch commands. The two or more segmented branches follow a pre-branching task segmentation that includes the pre-branching task. The task manager circuit transfers the task from the selected segmented branch among these segmented branches to the data processor circuit for execution of the task. Attached Figure Description

[0008] Figure 1 It is a high-level diagram of an electronic device according to the implementation plan.

[0009] Figure 2 This is a block diagram illustrating components in an electronic device according to one embodiment.

[0010] Figure 3 This is a block diagram illustrating a neural processor circuit according to one embodiment.

[0011] Figure 4 This is a block diagram of a neural engine in a neural processor circuit according to one implementation scheme.

[0012] Figure 5 This is a block diagram of a planar engine in a neural processor circuit according to one implementation scheme.

[0013] Figure 6 This is a diagram illustrating the programming of the grating and data flow control circuitry, components for configuring neural processor circuitry, according to one embodiment.

[0014] Figure 7 This is a schematic block diagram illustrating a neural network represented by a task list according to one implementation scheme.

[0015] Figure 8 This is a timing diagram illustrating the execution of a task in a neural processor circuit according to one embodiment.

[0016] Figure 9 It is a block diagram of the data control circuit in a data processor circuit according to one implementation scheme.

[0017] Figure 10This is a block diagram of a neural task manager in a neural processor circuit according to one implementation scheme.

[0018] Figure 11A This is a block diagram illustrating one or more neural network representations according to one implementation scheme.

[0019] Figure 11B It is a block diagram illustrating one or more task sequences according to an implementation scheme.

[0020] Figure 12 This is a block diagram illustrating a task descriptor according to one implementation scheme.

[0021] Figure 13 This is a block diagram illustrating a branching process according to one implementation scheme.

[0022] Figure 14 This is a flowchart illustrating an exemplary process for performing branched neural processing operations according to one embodiment.

[0023] For illustrative purposes only, the accompanying drawings and detailed descriptions depict various non-limiting embodiments. Detailed Implementation

[0024] Reference will now be made in detail to the embodiments, examples of which are shown in the accompanying drawings. Numerous specific details are shown in the following detailed description to provide a full understanding of the various described embodiments. However, the embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the various aspects of the embodiments.

[0025] Embodiments of this disclosure relate to a neural processor including task manager circuitry that selects a branch from two or more branching options based on the output of a previous task. The task manager enqueues the selected task branch after determining the result of the previous task. The selected task branch can be assigned to a different neural network than the one that includes the branch-determining task. Branch determination at the neural processor level reduces the transition latency from one neural network to another, thereby accelerating the operation of the neural processor.

[0026] Exemplary electronic devices

[0027] This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functionalities such as a personal digital assistant (PDA) and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Devices, Apple Equipment and Device. Alternatively, other portable electronic devices, such as wearable devices, laptops, or tablets, may be used. In some embodiments, the device is not a portable communication device, but a desktop computer or other computing device not designed for portable use. In some embodiments, the disclosed electronic device may include a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). The following is combined with… Figure 1 The described example electronic device (e.g., device 100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0028] Figure 1 This is a high-level diagram of an electronic device 100 according to one embodiment. Device 100 may include one or more physical buttons, such as a "home" button or a menu button 104. Menu button 104 is used, for example, to navigate to any application in a set of applications running on device 100. In some embodiments, menu button 104 includes a fingerprint sensor for recognizing a fingerprint on menu button 104. The fingerprint sensor can be used to determine whether the finger on menu button 104 has a fingerprint that matches a fingerprint stored for unlocking device 100. Alternatively, in some embodiments, menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touchscreen.

[0029] In some embodiments, device 100 includes a touchscreen 150, a menu button 104, a push-button 106 for powering the device on / off and for locking the device, a volume control button 108, a subscriber identity module (SIM) card slot 110, a headset jack 112, and a docking / charging external port 124. The push-button 106 can be used to power the device on / off by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 includes various components, including but not limited to memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include more than one type of image sensor 164. Each type may include more than one image sensor 164. For example, one type of image sensor 164 may be a camera, and another type of image sensor 164 may be an infrared sensor for facial recognition performed by one or more machine learning models stored in device 100. Device 100 may include Figure 1 Components not shown include, for example, an ambient light sensor, a dot projector, and a floodlight illuminator for supporting facial recognition.

[0030] Device 100 is merely one example of an electronic device, and device 100 may have more or fewer components than those listed above, some of which may be combined into a single component or have different configurations or arrangements. The various components of device 100 listed above are embodied in hardware, software, firmware, or combinations thereof, including one or more signal processing and / or application-specific integrated circuits (ASICs).

[0031] Figure 2 This is a block diagram illustrating components in device 100 according to one embodiment. Device 100 is capable of performing various operations, including implementing one or more machine learning models. For this and other purposes, device 100 may include an image sensor 202, a system-on-a-chip (SOC) component 204, system memory 230, permanent memory (e.g., flash memory) 228, a motion sensor 234, and a display 216, among other components. Figure 2 The components shown are merely illustrative. For example, device 100 may include... Figure 2Other components not shown (such as speakers or microphones). Additionally, some components (such as motion sensor 234) may be omitted from device 100.

[0032] Image sensor 202 is a component for capturing image data and can be implemented as, for example, a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, camera, camcorder, or other device. Image sensor 202 generates raw image data, which is sent to SOC component 204 for further processing. In some embodiments, the image data processed by SOC component 204 is displayed on display 216, stored in system memory 230, permanent memory 228, or transmitted to a remote computing device via a network connection. The raw image data generated by image sensor 202 may be a Bayer color kernel array (CFA) pattern.

[0033] Motion sensor 234 is a component or set of components used to sense the motion of device 100. Motion sensor 234 can generate sensor signals indicating the orientation and / or acceleration of device 100. The sensor signals are sent to SOC component 204 for various operations, such as turning on device 100 or rotating an image displayed on display 216.

[0034] Display 216 is a component for displaying images generated by SOC component 204. Display 216 may include, for example, a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) device. Based on data received from SOC component 204, display 216 may display various images, such as menus, selected operating parameters, images captured by image sensor 202 and processed by SOC component 204, and / or other information (not shown) received from the user interface of device 100.

[0035] System memory 230 is a component used to store instructions executed by SOC component 204 and to store data processed by SOC component 204. System memory 230 can be embodied in any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or combinations thereof.

[0036] Permanent memory 228 is a component used to store data in a non-volatile manner. Even when power is unavailable, permanent memory 228 retains the data. Permanent memory 228 may be embodied as read-only memory (ROM), flash memory, or other non-volatile random access memory devices. Permanent memory 228 stores the operating system and various software applications of device 100. Permanent memory 228 may also store one or more machine learning models, such as regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional networks (CNNs), recurrent networks (RNNs), autoencoders, and long short-term memory (LSTMs). Machine learning models can be standalone models that work with neural processor circuitry 218 and various software applications or sensors of device 100. Machine learning models can also be part of software applications. Machine learning models can perform various tasks, such as face recognition, image classification, object, concept and information classification, speech recognition, machine translation, voice recognition, voice command recognition, text recognition, text and context analysis, other natural language processing, prediction, and suggestion.

[0037] Various machine learning models stored in device 100 can be fully trained, untrained, or partially trained to allow device 100 to enhance or continue training the machine learning models as device 100 is used. The operation of the machine learning models includes various computations used to train the models and to determine runtime results using the models. For example, in one scenario, device 100 captures a user's facial image and uses that image to further improve the machine learning model used to lock or unlock device 100.

[0038] SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing procedures. SOC component 204 may include, among other sub-components, an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a memory controller 226, and a bus 232 connecting these sub-components. SOC component 204 may include... Figure 2 The sub-components shown have more or fewer sub-components.

[0039] ISP 206 is a circuit that performs each stage of the image processing pipeline. In some implementations, ISP 206 may receive raw image data from image sensor 202 and process the raw image data into a form available to other sub-components of SOC component 204 or components of device 100. ISP 206 may perform various image manipulation operations, such as image panning, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations.

[0040] CPU 208 can be implemented using any suitable instruction set architecture and can be configured to execute instructions defined in that instruction set architecture. CPU 208 can be a general-purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, RISC, ARM, or MIPS ISA, or any other suitable ISA. Although Figure 2 A single CPU is shown, but SOC component 204 may include multiple CPUs. In a multiprocessor system, each CPU may collectively implement the same ISA, but this is not required.

[0041] The graphics processing unit (GPU) 220 is a graphics processing circuit for executing graphics data. For example, the GPU 220 may render objects to be displayed in a frame buffer (e.g., a frame buffer that includes pixel data for the entire frame). The GPU 220 may include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of some graphics operations.

[0042] Neural processor circuit 218 is a circuit that performs various machine learning operations based on computations including multiplication, addition, and accumulation. Such computations can be arranged, for example, to perform various types of tensor multiplication, such as tensor products and convolutions of input data and kernel data. Neural processor circuit 218 is a configurable circuit that performs these operations in a fast and efficient manner, while alleviating the resource-intensive operations associated with CPU 208 and neural network operations. Neural processor circuit 218 can receive input data from sensor interface 212, image signal processor 206, permanent memory 228, system memory 230, or other sources such as network interface 210 or GPU 220. The output of neural processor circuit 218 can be provided to various components of device 100, such as image signal processor 206, system memory 230, or CPU 208, for various operations. The structure and operation of neural processor circuit 218 are referenced below. Figure 3 Detailed description.

[0043] Network interface 210 is a sub-component that supports the exchange of data between device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data may be received from other devices via network interface 210 and stored in system memory 230 for subsequent processing (e.g., via a back-end interface to image signal processor 206) and display. Networks may include, but are not limited to, local area networks (LANs) (e.g., Ethernet or corporate networks) and wide area networks (WANs). Image data received via network interface 210 may be processed by ISP 206.

[0044] Sensor interface 212 is a circuit used to communicate with motion sensor 234. Sensor interface 212 receives sensor information from motion sensor 234 and processes the sensor information to determine the orientation or movement of device 100.

[0045] Display controller 214 is a circuit used to send image data to be displayed on display 216. Display controller 214 receives image data from ISP 206, CPU 208, graphics processor or system memory 230, and processes the image data into a format suitable for display on display 216.

[0046] The memory controller 222 is circuitry for communicating with the system memory 230. The memory controller 222 can read data from the system memory 230 for processing by the ISP 206, CPU 208, GPU 220, or other sub-components of the SOC component 204. The memory controller 222 can also write data to the system memory 230 received from various sub-components of the SOC component 204.

[0047] The video encoder 224 is hardware, software, firmware, or a combination thereof, used to encode video data into a format suitable for storage in permanent memory 128 or to pass data to network interface 210 for transmission over a network to another device.

[0048] In some implementations, one or more sub-components of SOC component 204, or some functions of these sub-components, may be executed by software components that run on neural processor circuitry 218, ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, permanent memory 228, or in another device that communicates with device 100 via network interface 210.

[0049] Example Neural Processor Circuit

[0050] The neural processor circuit 218 is a programmable circuit that performs machine learning operations on the input data of the neural processor circuit 218. The machine learning operations may include different computations for training a machine learning model and for performing inference or prediction based on the trained machine learning model.

[0051] Taking a CNN as an example of a machine learning model, CNN training can include forward propagation and backward propagation. A neural network can include an input layer, an output layer, and one or more intermediate layers, which may be called hidden layers. Each layer can include one or more nodes, which may be fully or partially connected to other nodes in adjacent layers. In forward propagation, the neural network performs computations in the forward direction based on the output of the previous layer. The operations of a node can be defined by one or more functions. Functions defining node operations can include various computational operations, such as data convolution using one or more kernels, layer pooling, tensor multiplication, etc. Functions can also include activation functions that adjust the weights of the node's output. Nodes in different layers can be associated with different functions. For example, a CNN may include one or more convolutional layers mixed with pooling layers and followed by one or more fully connected layers.

[0052] Each function (including the kernel) in a machine learning model can be associated with different coefficients that can be adjusted during training. Additionally, individual nodes in a neural network can also be associated with activation functions that determine the weights of the node's output during forward propagation. Common activation functions include step functions, linear functions, sigmoid functions, hyperbolic tangent functions (tanh), and rectified linear unit functions (ReLU). After a batch of training sample data passes through the neural network in forward propagation, the result is compared to the training labels of the training samples to calculate the network's loss function, which represents the network's performance. Furthermore, the neural network performs backpropagation by adjusting the coefficients in the individual functions using coordinate descent techniques such as stochastic coordinate descent (SGD) to improve the value of the loss function.

[0053] During training, device 100 may use neural processor circuitry 218 to perform all or some of the operations in forward and backward propagation. Multiple rounds of forward and backward propagation may be performed by neural processor circuitry 218 alone or in coordination with other processors such as CPU 208, GPU 220, and ISP 206. Training may be completed when the loss function no longer improves (e.g., the machine learning model has converged) or after a predetermined number of rounds for a specific training sample. While using device 100, it may continue to collect additional training samples for the neural network.

[0054] To make predictions or inferences, device 100 may receive one or more input samples. Neural processor circuitry 218 may take the input samples and perform forward propagation to determine one or more results. Input samples may be images, speech, text files, sensor data, or other data.

[0055] In machine learning, data and functions (e.g., input data, kernels, functions, layer outputs, gradient data) can be stored and represented by one or more tensors. Common operations related to the training and runtime of machine learning models can include tensor product, tensor transpose, element-wise tensor operations, convolution, application of activation functions, automatic differentiation to determine gradients of values ​​in tensors, statistics and aggregations (e.g., mean, variance, standard deviation), tensor rank and size manipulation, etc.

[0056] While the training and runtime of a neural network have been discussed as an example, the neural processor circuit 218 can also be used to operate on other types of machine learning models, such as kernel SVMs. For simplicity, this disclosure describes the operation of a neural network, but these operations can also be used on other types of machine learning models.

[0057] See Figure 3 The exemplary neural processor circuitry 218 may, among other components, include a neural task manager 310, a plurality of neural engines 314A to 314N (collectively referred to below as "a plurality of neural engines 314" and also individually as "neural engine 314"), kernel direct memory access (DMA) 324, data processor circuitry 318, data processor DMA 320, planar engine 340, and neural processor (NP) controller 350. The neural processor circuitry 218 may include... Figure 3 The fewer components shown or including Figure 3 Additional components not shown.

[0058] Each neural engine in neural engine 314 performs machine learning computational operations in parallel. Depending on the workload, the entire group of neural engines 314 may be operating, or only a subset of neural engines 314 may be operating, while the remaining neural engines 314 are placed in a power-saving mode to conserve power. Each neural engine in neural engine 314 includes components for storing one or more kernels, for performing multiplication and accumulation operations, and for post-processing to generate output data 328, as described below. Figure 4 Detailed description. The Neural Engine 314 is specifically designed to perform computationally intensive operations, such as convolution and tensor product operations. Convolution operations can include different types of convolution, such as cross-channel convolution (convolution that sums values ​​from different channels), communication-by-communication convolution, and transposed convolution.

[0059] Planar engine 340 can specialize in performing simpler computational operations, the speed of which may depend primarily on the input and output (I / O) speed of data transfer rather than the computational speed within planar engine 340. These computational operations may be referred to as I / O-constrained computations and are also referred to herein as “non-convolutional operations.” In contrast, neural engine 314 can focus on complex computations, such as convolutional operations, the speed of which may depend primarily on the computational speed within each neural engine 314. For example, planar engine 340 is efficient for performing operations within a single channel, while neural engine 314 is efficient for performing operations across multiple channels that may involve heavy accumulation of data. Computations with I / O constraints using neural engine 314 cannot be efficient in both speed and power consumption. In one implementation, the input data may be a tensor with a rank greater than three (e.g., having three or more dimensions). One set of dimensions (two or more) in the tensor may be referred to as a plane, and another dimension may be referred to as a channel. Neural engine 314 may utilize kernels to convolve the data in the planes of the tensor and accumulate the results of convolutions in different planes across different channels. On the other hand, the planar engine 340 can be dedicated to operations within a plane.

[0060] The circuitry of the planar engine 340 can be programmed to operate in one of several modes, including pooling mode, element-wise mode, and shrinking mode. In pooling mode, the planar engine 340 shrinks the spatial size of the input data. In element-wise mode, the planar engine 340 generates the output derived from element-wise operations on one or more inputs. In shrinking mode, the planar engine 340 shrinks the rank of a tensor. For example, a rank-5 tensor can be shrunk to a rank-2 tensor, or a rank-3 tensor can be shrunk to a rank-0 tensor (e.g., a scalar). See below for reference. Figure 5 The operation of the planar engine 340 will be discussed in further detail.

[0061] The neural task manager 310 manages the overall operation of the neural processor circuitry 218. The neural task manager 310 may receive a list of tasks from a compiler executed by the CPU 208, store tasks in its task queue, select tasks to be executed, and send task commands to other components of the neural processor circuitry 218 for executing the selected tasks. Data may be associated with task commands indicating the type of operation to be performed on the data. The data of the neural processor circuitry 218 includes input data transferred from another source, such as system memory 230, and data generated by the neural processor circuitry 218 in previous operating cycles. Each dataset may be associated with a task command specifying the type of operation to be performed on the data. The neural task manager 310 may also perform task switching upon detecting events such as receiving instructions from the CPU 208. In one or more embodiments, the neural task manager 310 sends raster information to components of the neural processor circuitry 218 to enable each of these components to track, retrieve, or process appropriate portions of the input data and kernel data. For example, the neural task manager 310 may include registers storing information about the size and rank of the datasets for processing by the neural processor circuitry 218. Although in Figure 3 The neural task manager 310 is shown as part of the neural processor circuitry 218, but the neural task manager 310 may be a component external to the neural processor circuitry 218.

[0062] Kernel DMA 324 is a read circuit that retrieves kernel data from a source (e.g., system memory 230) and sends kernel data 326A to 326N to each neural engine in neural engine 314. Kernel data represents information from which kernel elements can be extracted. In one embodiment, the kernel data may be a compressed format that is decompressed at each neural engine in neural engine 314. Although in some cases the kernel data provided to each neural engine in neural engine 314 may be the same, in most cases the kernel data provided to each neural engine in neural engine 314 is different. In one embodiment, the direct memory access nature of kernel DMA 324 allows kernel DMA 324 to directly retrieve and write data from the source without the involvement of CPU 208.

[0063] Data processor circuitry 318 manages the data flow and task performance of neural processor circuitry 218. Data processor circuitry 318 may include data control circuitry 332 and buffer 334. Buffer 334 is a temporary storage device used to store: data associated with the operation of neural processor circuitry 218, such as input data transferred from system memory 230 (e.g., data from a machine learning model); and other data generated within neural processor circuitry 218. Input data can be transferred from system memory 230. The data stored in data processor circuitry 318 may include different subsets, which are sent to various downstream components, such as neural engine 314 and planar engine 340.

[0064] In one embodiment, buffer 334 is implemented as a non-transitory memory accessible to neural engine 314 and planar engine 340. Buffer 334 may store: input data 322A to 322N (also referred to herein as “neural input data”) for feeding to the corresponding neural engines 314A to 314N and input data 342 (also referred to herein as “planar input data”) for feeding to the planar engine 340; and output data 328A to 328N (also referred to herein as “neural output data”) from each neural engine 314A to 314N and output data 344 from the planar engine 340 (also referred to herein as “planar output data”) for feeding back to one or more neural engines 314 or planar engine 340 or to a target circuit (e.g., system memory 230). Buffer 334 may also store input data 342 and output data 344 of the planar engine 340 and allows data exchange between neural engine 314 and planar engine 340. For example, one or more output data 328A to 328N of neural engine 314 can be used as planar input data 342 to planar engine 340. Similarly, planar output data 344 of planar engine 340 can be used as input data 322A to 322N of neural engine 314. The input to neural engine 314 or planar engine 340 can be any data stored in buffer 334. For example, in each operation loop, the source dataset from which one of the engines obtains input can be different. The input to an engine can be the output of the same engine in a previous loop, the output of a different engine, or any other suitable source dataset stored in buffer 334. Furthermore, the dataset in buffer 334 can be partitioned and sent to different engines for different operations in the next operation loop. Two datasets in buffer 334 can also be combined for the next operation.

[0065] The data control circuit 332 of the data processor circuit 318 controls the data exchange between the neural engine 314 and the planar engine 340. The operation of the data processor circuit 318 and other components of the neural processor circuit 218 is coordinated so that input data and intermediate data stored in the data processor circuit 318 can be reused across multiple operations at the neural engine 314 and the planar engine 340, thereby reducing the data transfer to and from the system memory 230. Data control circuitry 332 may perform one or more of the following operations: (i) monitor the size and rank of the data being processed by neural engine 314 and planar engine 340 (e.g., the data may be one or more tensors), (ii) determine which subsets of data are transmitted to neural engine 314 or planar engine 340 based on task commands associated with different subsets of data, (iii) determine the manner in which data is transmitted to neural engine 314 and planar engine 340 (e.g., data processor circuitry 318 may operate in broadcast mode, where the same data is fed to multiple input channels of neural engine 314 such that multiple or all neural engines 314 receive the same data, or may operate in unicast mode, where different neural engines 314 receive different data), and (iv) transmit configuration commands to planar engine 340 to instruct planar engine 340 to program itself for operation in one of multiple operating modes. References below... Figure 9 Describe the details of the data control circuit 332.

[0066] The data stored in the buffer 334 of the neural processor circuit 218 may be a portion of image data, histograms of orientation gradient (HOG) data, audio data, metadata, output data 328 of the previous loop of the neural engine 314, and other processed data received from other components of the SOC component 204.

[0067] The data processor DMA 320 includes read circuitry that receives a portion of input data from a source (e.g., system memory 230) for storage in a buffer 334, and write circuitry that forwards the data from the buffer 334 to a target component (e.g., system memory). In one embodiment, the direct memory access nature of the data processor DMA 320 allows the data processor DMA 320 to directly acquire and write data from a source (e.g., system memory 230) without the involvement of the CPU 208. The buffer 334 may be a direct memory access buffer that stores data of the machine learning model of device 100 without the involvement of the CPU 208.

[0068] The neural processor (NP) controller 350 is a control circuit that performs various operations to control the overall operation of the neural processor circuit 218. The NP controller 350 can connect to the CPU 208 and the program unit of the neural processor circuit 218 and perform stewardship operations by setting registers in its components. The NP controller 350 can also initialize the components in the neural processor circuit 218 when the neural processor circuit 218 is turned on.

[0069] Example Neural Engine Architecture

[0070] Figure 4 This is a block diagram of a neural engine 314 according to one embodiment. The neural engine 314 is a circuit that performs various operations to facilitate machine learning, such as convolution, tensor product, and other operations that may involve large computational loads. To this end, the neural engine 314 receives input data 322, performs a multiplication-accumulation operation (e.g., a convolution operation) on the input data 322 based on stored kernel data, performs further post-processing operations on the result of the multiplication-accumulation operation, and generates output data 328. The input data 322 and / or output data 328 of the neural engine 314 may be single-channel or span multiple channels.

[0071] The neural engine 314 may include, among other components, an input buffer circuit 402, a computational core 416, a neural engine (NE) control 418, a kernel extraction circuit 432, an accumulator 414, and an output circuit 424. The neural engine 314 may include... Figure 4 The fewer components shown or including Figure 4 Other components not shown.

[0072] Input buffer circuit 402 is a circuit that stores a subset of data received from a source when a subset of data for neural processor circuit 218 is received. The source may be data processor circuit 318, planar engine 340, or another suitable component. Input buffer circuit 402 sends an appropriate portion 408 of data for the current task or processing loop to computational core 416 for processing. Input buffer circuit 402 may include shifter 410 that shifts the read position of input buffer circuit 402 to change the portion 408 of data sent to computational core 416. By changing the portion of the input data provided to computational core 416 via shifting, neural engine 314 can perform multiplication and accumulation on different portions of the input data based on a smaller number of read operations. In one or more embodiments, the data for neural processor circuit 218 includes differential convolution groups and / or input channel data.

[0073] Kernel extraction circuit 432 is a circuit that receives kernel data 326 from kernel DMA 324 and extracts kernel coefficients 422. In one embodiment, kernel extraction circuit 432 references a lookup table (LUT) and uses a mask to reconstruct the kernel from compressed kernel data 326 based on the LUT. The mask indicates the locations of unfilled zeros and the remaining locations of unfilled numbers in the reconstructed kernel. The kernel coefficients 422 of the reconstructed kernel are sent to compute core 416 to populate registers in the multiply-accumulate (MAD) circuitry of compute core 416. In other embodiments, kernel extraction circuit 432 receives kernel data in uncompressed format and determines the kernel coefficients without referencing a LUT or using a mask.

[0074] The computational core 416 is a programmable circuit that performs computational operations. For this purpose, the computational core 416 may include MAD circuits MAD0 to MADN and a post-processor 428. Each of the MAD circuits MAD0 to MADN stores an input value in an input data portion 408 and a corresponding kernel coefficient in a kernel coefficient 422. In each of the MAD circuits, the input value and the corresponding kernel coefficient are multiplied to generate a processed value 412.

[0075] Accumulator 414 is a memory circuit that receives and stores processed values ​​412 from the MAD circuit. The processed values ​​stored in accumulator 414 can be sent back as feedback information 419 for further multiplication and addition operations at the MAD circuit, or sent to post-processor 428 for post-processing. Accumulator 414, combined with the MAD circuit, forms multiplication accumulator (MAC) 404. In one or more embodiments, accumulator 414 may have sub-units, each of which sends data to a different part of neural engine 314. For example, during a processing cycle, data stored in a first sub-unit of accumulator 414 is sent to the MAC circuit, while data stored in a second sub-unit of accumulator 414 is sent to post-processor 428.

[0076] Postprocessor 428 is circuitry that performs further processing on the value 412 received from accumulator 414. Postprocessor 428 may perform operations including, but not limited to, applying linear functions (e.g., rectified linear unit (ReLU)), normalizing cross-correlation (NCC), combining the results of neural operations on 8-bit data into 16-bit data, and local response normalization (LRN). The result of such operations is output from postprocessor 428 as a processed value 417 to output circuitry 424. In some embodiments, processing at postprocessor 428 is bypassed. For example, data in accumulator 414 may be sent directly to output circuitry 424 for access by other components of neural processor circuitry 218.

[0077] The NE control 418 controls the operation of other components of the neural engine 314 based on the operating mode and parameters of the neural processor circuit 218. Depending on different operating modes (e.g., group convolution mode or non-group convolution mode) or parameters (e.g., the number of input channels and the number of output channels), the neural engine 314 can operate on different input data in different sequences, return different values ​​from the accumulator 414 to the MAC circuit, and perform different types of post-processing operations at the post-processor 428. To configure the components of the neural engine 314 to operate in a desired manner, the NE control 418 sends task commands, which can be included in the information 419, to the components of the neural engine 314. The NE control 418 may include a raster 430 that tracks the current task or processing loop being processed at the neural engine 314.

[0078] Input data is typically divided into smaller data slices for parallel processing across multiple neural engines 314 or between neural engine 314 and planar engine 340. A set of data used for convolution operations may be called a convolution group, which can be divided into multiple smaller units. The hierarchical structure of these smaller units (parts of data) can be convolution groups, slices, blocks, working units, output channel groups, input channels (Cin), sub-Cin for input strides, etc. For example, a convolution group can be divided into slices; slices can be divided into blocks; blocks can be divided into working units; and so on. In the context of neural engine 314, a working unit can be a portion of the input data, such as data processed by planar engine 340 or data processed in a previous loop of neural engine 314, having a size that produces the output value of accumulator 414 suitable for neural engine 314 during a single loop of computation kernel 416. In one case, the size of each working unit is 256 bytes. In such implementations, for example, a working unit may be shaped into one of a 16×16, 32×8, 64×4, 128×2, or 256×1 dataset. In the context of the planar engine 340, a working unit may be (i) a portion of the input data, (ii) data from the neural engine 314, or (iii) data from a previous loop of the planar engine 340, which may be processed simultaneously at the planar engine 340.

[0079] Rasterizer 430 performs operations associated with dividing input data into smaller units (parts) and regulates the processing of these smaller units via MAC 404 and accumulator 414. Rasterizer 430 maintains the size and rank (e.g., group, working unit, input channel, output channel) of the input / output data portions and instructs components of neural processor circuitry 218 to correctly process the portions of input data. For example, rasterizer 430 operates shifter 410 in input buffer circuitry 402 to forward the correct portion 408 of input data to MAC 404 and send the completed output data 328 to data buffer 334. Other components of neural processor circuitry 218 (e.g., kernel DMA 324, data processor DMA 320, data buffer 334, planar engine 340) may also have corresponding rasterizers for monitoring the division of input data and the parallel computation of the various portions of input data in different components.

[0080] The output circuit 424 receives the processed value 417 from the post-processor 428 and interacts with the data processor circuit 318 to store the processed value 417 in the data processor circuit 318. For this purpose, the output circuit 424 may issue output data 328 in an order or format different from the order or format in which the processed value 417 is processed in the post-processor 428.

[0081] Components in the Neural Engine 314 can be configured by the NE control 418 and the Neural Task Manager 310 during the configuration cycle. To this end, the Neural Task Manager 310 sends configuration information to the Neural Engine 314 during the configuration cycle. Configurable parameters and modes may include, but are not limited to, the mapping between input data elements and kernel elements, the number of input channels, the number of output channels, the execution of output strides, and enabling / selecting post-processing operations at the post-processor 428.

[0082] Example planar engine architecture

[0083] Figure 5This is a block diagram of a planar engine 340 according to one embodiment. The planar engine 340 is a separate circuit from the neural engine 314 and can be programmed to perform in different operating modes. For example, the planar engine 340 can operate in pooling mode, shrinking mode, gain-biased mode, and element-wise mode. Pooling mode shrinks the spatial size of the data, shrinking mode shrinks the rank of the tensor, gain-biased mode provides a one-way addition of bias and scaling by a scaling factor, and element-wise mode includes element-wise operations. For this purpose, the planar engine 340 may include, among other components, a first format converter 502, a first filter 506 (also referred to herein as "multimode horizontal filter 506"), a line buffer 510, a second filter 514 (also referred to herein as "multimode vertical filter 514"), a post-processor 518, a second format converter 522, and a planar engine (PE) control 530 (including a rasterizer 540). The planar engine 340 may include fewer components or may include... Figure 5 Other components not shown in A. Each component in the planar engine 340 may be implemented as a circuit or a circuit combined with firmware or software.

[0084] Input data 342 for the planar engine 340 can be obtained from one or more source datasets stored in the data processor circuitry 318. If the dataset to be processed by the planar engine 340 is larger than the working unit of data that the planar engine 340 can process simultaneously, such a dataset can be segmented into multiple working units for use as input data 342 to be read into the planar engine 340. Depending on the mode of the planar engine 340, the input data 342 may include data from one or more source datasets. The source datasets described herein refer to different data stored in the neural processor circuitry 218 for processing. Different components of the neural processor circuitry 218 can generate or transfer data stored in the data processor circuitry 318. For example, the neural engine 314, the planar engine 340 (which generated data in a previous operating loop), and the system memory 230 can generate or transfer different datasets stored in different memory locations of the data processor circuitry 318. Each source dataset can represent a different tensor. During the operating loop of the planar engine 340, different source datasets can be obtained together as input data 342. For example, in element-wise mode involving the addition of two different tensors to derive the tensor, input data 342 may include data from two different source datasets, each providing a separate tensor. In other modes, a single source dataset may provide input data 342. For example, in pooling mode, input data 342 may be obtained from a single source dataset.

[0085] The first format converter 502 is a circuit that performs one or more format conversions on input data 342 in one format (e.g., a format for use in the buffer 334) to another form for processing in subsequent parts of the planar engine 340. Such format conversions may include, etc., applying a ReLU function to one or more values ​​of input data 342, converting one or more values ​​of input data 342 to their absolute values, transposing a tensor included in the source, applying a gain to one or more values ​​of input data 342, biasing one or more values ​​of input data 342, normalizing or denormalizing one or more values ​​of input data 342, converting floating-point numbers to signed or unsigned numbers (or vice versa), logarithmicizing, and changing the size of a tensor, such as by broadcasting the tensor's values ​​in one or more dimensions to expand the tensor's rank. The converted input data 342 and the unconverted input data 342 to the planar engine 340 are collectively referred to herein as "versions of input data".

[0086] The first filter 506 is a circuit that performs a filtering operation in one direction. For this purpose, the first filter 506 may include an adder, a comparator, and a multiplier, among other components. The filtering performed by the first filter 506 may be, for example, averaging, selecting a maximum value, or selecting a minimum value. When averaging, an adder is used to sum the values ​​of the input data 342, and a weighting factor may be applied using a multiplier to the sum to obtain an average value. When selecting a maximum or minimum value, a comparator may be used instead of an adder and a multiplier to select the value.

[0087] Row buffer 510 is a memory circuit for storing results, such as one or more intermediate data obtained from the first filter 506 or the second filter 514. Row buffer 510 can store values ​​from different rows and allows access from the second filter 514 or other downstream components to obtain intermediate data for further processing. In some modes, row buffer 510 is bypassed. Row buffer 510 may also include logic circuitry for performing additional operations beyond simply storing intermediate data. For example, row buffer 510 includes adder circuitry 512, which, combined with memory components, enables row buffer 510 to act as an accumulator that aggregates data generated from the results of the first filter 506 or the second filter 514 to store aggregated data in a separate, non-reduced dimension.

[0088] Similar to the first filter 506, the second filter 514 performs filtering operations, but in a different direction than the first filter 506. For this purpose, the second filter 514 may include adders, comparators, and multipliers, among other components. In pooling mode, the first filter 506 performs filtering operations in the first dimension, while the second filter 514 performs filtering operations in the second dimension. In other modes, the first filter 506 and the second filter 514 may operate differently. In reduction mode, for example, the first filter 506 performs element-wise operations, while the second filter 514 acts as a reduction tree to aggregate data values.

[0089] Postprocessor 518 is circuitry that performs further processing on values ​​obtained from other upstream components. Postprocessor 518 may include dedicated circuitry that is efficient for performing certain types of mathematical calculations that might be inefficient using general-purpose computing circuitry. The operations performed by postprocessor 518 may include, among others, performing square root operations and reciprocals of values ​​in reduction mode. Postprocessor 518 may be bypassed in other operating modes.

[0090] The second format converter 522 is circuitry that converts the results of previous components in the planar engine 340 from one format to another format for output data 344. Such format conversions may include, for example, applying a ReLU function to the results, transposing the resulting tensor, normalizing or denormalizing one or more values ​​of the results, and other digital format conversions. Output data 344 may be stored in the data processor circuitry 318 as output of the neural processor circuitry 218 or as input to other components of the neural processor circuitry 218 (e.g., the neural engine 314).

[0091] The PE control 530 is a circuit that controls the operation of other components in the planar engine 340 based on the operating modes of the planar engine 340. Depending on the operating mode, the PE control 530 programs registers associated with different components in the planar engine 340, causing the programmed components to operate in a certain way. The pipeline of connections between components or between components in the planar engine 340 can also be reconfigured. In pooling mode, for example, data processed by the first filter 506 can be stored in the row buffer 510 and then read by the second filter 514 for further filtering. However, in shrinking mode, data is processed by the first filter 506, then by the second filter 514, and then accumulated in the row buffer 510, which is programmed as an accumulator. In element-wise mode, the row buffer 510 can be bypassed.

[0092] The PE control 530 also includes a rasterizer 540 that tracks the current task or processing loop being processed at the planar engine 340. The rasterizer 540 is circuitry that tracks units or portions of input data and / or loops used to process input data in the planar engine 340. The rasterizer 540 controls the acquisition of portions of the planar engine 340 in each operating cycle and monitors the size and rank of each portion processed by the planar engine 340. For example, smaller portions of the dataset may be acquired as input data 342 in rasterizer order for processing at the planar engine 340 until all portions of the source dataset have been processed. While acquiring portions, the rasterizer 540 monitors the coordinates of the portions in the dataset. The way the dataset is segmented into input data 342 for processing at the planar engine 340 can be different from how the dataset is segmented into input data 328 for processing at the neural engine 314.

[0093] The dataset to be processed at the planar engine 340 may be larger than the capacity of the planar engine 340, which can process data in a single operation loop. In this case, the planar engine 340 acquires different portions of the dataset as input data 342 in multiple operation loops. The acquired portions may partially overlap with previously acquired portions and / or the next portion to be acquired. In one embodiment, the overlapping data portions are acquired only once and reused to reduce the time and power consumption costs of the planar engine 340 in acquiring data.

[0094] Example compilation of Neural Task Manager and Task List

[0095] Figure 6 This diagram illustrates the programming of rasterizers 614, 622, 624 and data control circuitry 332 in components 314, 318, 322, 340 of a neural processor circuitry 218 according to one embodiment. To perform their functions, rasterizers 614, 622, 624 and data control circuitry 332 receive configuration data 610 instructing how input data and / or kernel data are segmented and processed by each component of the neural processor circuitry 218. This task information includes details about the current layer (e.g., the dimensions of the input and output data, the dimensions of the associated kernel, and the type of padding at input data boundaries). Rasterizers 614, 622, 624 and data control circuitry 332 may also receive constraints on their operation (e.g., whether tile widths are allowed or not allowed to exceed a threshold). The configuration data 610 sent to the data control circuit 332 may further include information about data dependencies and data conflicts, enabling the data control circuit 332 to coordinate the reading of input data from the data processor circuit 318 to the neural engine 314 and the planar engine 340, and to coordinate the writing of output data from the neural engine 314 and the planar engine 340 to the data processor circuit 318.

[0096] By providing rasterizers in different components of the neural processor circuit 218, the overhead of data transfer between components of the neural processor circuit 218 can be reduced. If a single central rasterizer were provided to control the different components of the neural processor circuit 218, kernel data, input data, and output data might need to be transferred between these components to identify associated locations in the task loop, such as convolutional groups, tiles, slices, working units, input channels, and output channels. By using distributed rasterizers, separate metadata is not required to transfer kernel data, input data, and output data between components of the neural processor circuit 218.

[0097] The neural network may include network layers or sublayers that are instantiated or implemented as a series of tasks performed by the neural processor circuit 218. Figure 7 This is a schematic block diagram illustrating a neural network 700 represented by a task list 704 according to one embodiment. The neural network 700 includes network layers (or sublayers) including convolutional layers C1, C2, and C3 (including sublayer C3). 00 C3 10 C3 11 C3 20 and C3 21 The neural network 700 consists of C4 and non-convolutional layers (e.g., pooling layers) P1, P2, P3, and P4. The neural network 700 is an example of a neural network architecture that can be instantiated by the neural processor circuit 218. That is, when the task in the neural network 700 is transformed into a task list 704 to become executable by the neural processor circuit 218. Other types of neural network architectures with different types of network layers or network hierarchical order can also be instantiated by the neural processor circuit 218.

[0098] Neural network 700 is transformed into task list 704 through a compiler process, for example, executed by CPU 208. Task list 704 includes a series of tasks, including neural engine tasks TC1 to TC4 (corresponding to convolutional layers C1 to C4) and planar engine tasks TP1 to TP5 (corresponding to pooling layers P1 to P5). Neural engine task TC3 is further divided into smaller neural engine tasks TC3. 00 To TC3 21 (corresponding to sublayer C3) 00 To C3 21 In some embodiments, the task list 704 is stored in a linked list format. In other embodiments, the task list 704 is stored as one or more segments. Each segment may be stored in an adjacency region of memory containing one or more tasks to be executed. (See reference...) Figure 11A The detailed structure and configuration of the segments will be discussed further. Although Figure 7The exemplary task list 704 shown is illustrated as a linearly linked task chain, but the neural engine tasks and planar engine tasks do not need to be executed in this sequence. Instead, to increase the efficiency of the neural processor circuitry 218, it is desirable to execute the planar engine tasks in parallel with the neural engine tasks, provided that data dependencies and data conflicts are resolved. In one or more embodiments, the task sequence between neural engine tasks and the task sequence between planar engine tasks are maintained as determined during the compiler process, but the sequence between planar engine tasks and neural engine tasks can be switched.

[0099] Each task is associated with a task descriptor, which defines the configuration of the neural processor circuitry 218 to perform the task. Each task may correspond to a single network layer of the neural network 700, a portion of the network layers of the neural network 700, or multiple network layers of the neural network 700. The neural processor circuitry 218 instantiates the neural network 700 by executing tasks from the task list 704 under the control of the neural task manager 310.

[0100] Asynchronous execution of neural engine tasks and planar engine tasks

[0101] Figure 8 This is a timing diagram illustrating the execution of a task in a neural processor circuit 218 according to one embodiment. In this example, the task begins in the sequence TC1, TP1, TC2, TC3, TP2, TP3, TP4, TP5, and TC4. This sequence is not related to... Figure 7 The task sequence in task list 704 is consistent. Because task TC3 is a long neural engine task, and planar engine 340 can perform operations in parallel with neural engine 314, planar engine tasks TP2 to TP4 are executed simultaneously with neural engine task TC3. By processing tasks TP2 to TP4 in parallel with task TC3, the data for task TP5 can be obtained faster than by executing the process in the sequence of task list 704.

[0102] although Figure 8 Adjacent neural engine tasks and adjacent planar engine tasks are illustrated as being separated by a time difference to facilitate interpretation; however, in practice, the execution times of adjacent neural engine tasks and adjacent planar engine tasks can overlap. Neural engine 314 and planar engine 340 can employ a pipelined processing architecture, where they can receive input data for one task while simultaneously producing output data for a previous task. Neural engine 314 and planar engine 340 can operate on data from different tasks simultaneously. For example, neural engine 314 can begin task TC3 before task TC2 is completed, and planar engine 340 can begin task TP3 before task TP2 is completed.

[0103] To address the data dependency issue, the data control circuit 332 controls the timing of sending neural input data and planar input data to the neural engine 314 and planar engine 340, respectively. For this purpose, the data control circuit 332 may include, among other components, features such as... Figure 9 The access enable circuit 910 and the raster 920 are shown. The data control circuit 332 may include... Figure 9 Other components not shown.

[0104] Access enable circuit 910 is a programmable circuit that selectively grants access to read data from or write data to buffer 334 of data processor circuit 318. Neural engine 314 and planar engine 340 can be configured such that their circuitry and components do not produce output data until input data is provided. Therefore, access enable circuit 910 can postpone the pending tasks of neural engine 314 or planar engine 340 by preventing neural engine 314 or planar engine 340 from accessing input data in buffer 334 until all dependent data for pending tasks in buffer 334 is available. Access enable circuit 910 can determine the data dependencies of a task by reading and analyzing dependency information included in task information entries corresponding to the task, and by referring to the state of the task tracked by rasterizer 920 to determine whether all dependent data is available in buffer 334. In this way, access enable circuit 910 can prevent the start of the next task, which depends on the output data of the previous task, until the output data of the previous task is stored in buffer 334 and available in the buffer.

[0105] Access enable circuitry 910 also prevents output data (generated by neural engine 314 and planar engine 340) from being written to buffer 334 or from input data from being read to resolve data conflict issues. For this purpose, access enable circuitry 910 may reference the state of operation indicated by rasterizer 920. Based on the indicated state, access enable circuitry 910 may prevent neural engine 314 or planar engine 340 from writing output data to buffer 334 until another operation or task is completed, or prevent neural engine 314 or planar engine 340 from reading input data from buffer 334 for the current task until at least a portion of the output data from the previous task is stored in buffer 334. Access enable circuitry 910 may also perform additional arbitration between any neural engine 314 and planar engine 340 to access buffer 334.

[0106] The rasterizer 920 is a circuit that tracks the current size of the data used for processing each task or processing loop at the data processor circuitry 318. The function and operation of the rasterizer 920 are essentially the same as described in the reference above. Figure 4 and Figure 6The same applies to the rasterizers explained in detail. In one or more embodiments, at a given time, rasterizer 920 may track a task different from the task being tracked by other rasterizers (e.g., rasterizer 620 and rasterizer 622).

[0107] Example Task Management

[0108] Figure 10 This is a block diagram illustrating a neural task manager 310 according to one embodiment. The neural task manager 310 manages tasks executed by neural processor circuitry 218 for one or more neural networks 700. Among other components, the neural task manager 310 may include a task arbitrator 1002, task queues 1004A to 1004N (collectively referred to hereinafter as “multiple task queues 1004” and also individually as “task queue 1004”), a task manager direct memory access (DMA) 1006, a fetch queue 1008, and a configuration queue 1010. The neural task manager 310 may include... Figure 10 Other components not shown. For each task, the neural task manager 310 may receive a task descriptor 1012 from the software compilation process. The task descriptor may define the configuration of the neural processor circuitry 218 to execute a corresponding neural engine task or a corresponding planar engine task. The neural task manager 310 transmits a version of the task descriptor 1012 (e.g., task descriptor 1012 or data configuration 1014 extracted from task descriptor 1012) to the data processor circuitry 318.

[0109] Task arbiter 1002 is a circuit or combination of circuitry and firmware that selects tasks from task queue 1004 for execution by neural processor circuitry 218. Task arbiter 1002 dequeues tasks from task queue 1004 and places them in configuration queue 1010. While a task is in the configuration queue, it is submitted for execution, and the neural processor circuitry prefetches input data and kernel data before the task is executed by other components of neural processor circuitry 218. For example, task arbiter 1002 can perform priority arbitration among multiple task queues 1004 and dequeue the task with the highest priority from task queue 1004.

[0110] The neural task manager 310 may include one or more task queues 1004. Each task queue 1004 is coupled to the CPU 208 and the task arbiter 1002. Each task queue 1004 may include first-in-first-out (FIFO) hardware for arranging network segments that can be stored in a memory location, such as system memory 230. An exemplary configuration of the network segments is described in... Figure 11AThe above is shown as segment 1150. Each segment in segment 1150 may include multiple tasks. Each task may refer to task descriptor 1012. Task descriptor 1012 specifies the configuration of neural processor circuitry 218 used to perform the task. Each task queue 1004 may also be associated with a priority parameter that defines the relative priority of task queue 1004.

[0111] Task Manager DMA 1006 is coupled to Task Arbiter 1002, System Memory 230, and Acquisition Queue 1008. Task Manager DMA 1006 includes read circuitry that receives a task descriptor 1012 from a source (e.g., System Memory 230) and stores the task descriptor 1012 in Acquisition Queue 1008. For example, Task Arbiter 1002 selects Task Queue 1004 based on its priority and controls Task Manager DMA 1006 to select the task descriptor 1012.

[0112] The fetch queue 1008 is a single entry queue that stores task descriptors 1012 for pending submissions of tasks to be executed. The fetch queue 1008 is coupled to the task manager DMA 1006 to receive task descriptors 1012 from system memory 230. The fetch queue 1008 provides task descriptors 1012 to the configuration queue 1010, or provides configuration data 1014 extracted from task descriptors 1012 to the configuration queue 1010.

[0113] Configuration queue 1010 holds configuration data 1014 submitted for execution of multiple tasks. While a task is in configuration queue 1010, kernel DMA 324 may retrieve kernel data from system memory 230 to store in kernel fetch circuitry 432 of neural engine 314, and data processor DMA 320 may retrieve input data from system memory 230 to store in buffer 334 of data processor circuitry 318. To execute a task, kernel fetch circuitry 432 provides prefetched kernel data to MAC 404 of neural engine 314, and data buffer 334 provides prefetched input data to MAC 404 of neural engine 314. Planar engine 340 also accesses data processor circuitry 318 to read its input data 342. In some embodiments, configuration queue 1010 may include multiple queues holding configuration data 1014 fetched from submitted task descriptors 1012.

[0114] Figure 11AThis is a block diagram illustrating one or more neural network representations 1100 according to one embodiment. Each neural network representation 1100 includes information, metadata, and tasks corresponding to the neural network. The neural network representation 1100 can be instantiated by CPU 208 for processing by neural processor circuitry 218. For example, neural network representation 1100 can be generated when CPU 208 compiles a neural network or any machine learning model (also referred to as a neural network for simplicity and for the purposes of discussion). CPU 208 determines the tasks to be performed and stores these tasks in the neural network representation 1100. The neural network representation 1100 can be stored in memory (e.g., system memory 230 or memory in data processor circuitry 318). At a given time, neural processor circuitry 218 can perform computations related to more than one neural network. For example, device 100 can run multiple software applications that use neural networks for various purposes. Thus, neural processor circuitry 218 can access multiple neural network representations 1100, and can sometimes switch operations between the various neural networks if neural task manager 310 determines that one neural network should take precedence over another.

[0115] The neural network representation 1100 may include a network descriptor 1110 and one or more segments 1130. The network descriptor 1110 may be stored in an adjacency region of memory, which stores information related to network-wide configuration and auxiliary tasks for context switching, which may be referred to as context switching task 1120. The memory storing the network descriptor 1110 may vary depending on the implementation. For example, in some implementations, the network descriptor 1110 is stored in system memory 230, while in other implementations, the network descriptor 1110 is stored within neural processor circuitry 218, such as in the memory location of data processor circuitry 318 or task manager 310. In some implementations, the information in the network descriptor 1110 is fixed when the CPU 208 compiles the neural network representation 1100. In those implementations, the neural processor circuitry 218 does not write to the network descriptor 1110.

[0116] The neural descriptor 1110 includes various fields that describe the network extent settings of the neural network representation 1100 and specify the data locations of the neural network. For example, the neural descriptor 1110 may include a network identifier (ID) 1102, one or more base address indices 1106, a counting parameter 1108, an external system memory address 1112, a notification parameter 1114, a wait parameter 1116, and a context switching task 1120. Some data fields in the network descriptor 1110 may be stored as bits in the form of a data bit header. Other data fields may be stored as separate objects in memory locations. For example, in one embodiment, fields 1102, 1112, 1106, and 1108 are data bits included in the header of the network descriptor 1110, and the context switching task 1120 takes the form of a task descriptor 1012, the configuration of which will refer to... Figure 12 Further discussion.

[0117] The various fields in network descriptor 1110 provide configuration information for neural network representation 1100. For example, network ID 1102 may be a unique identifier for the neural network. Network descriptor 1110 may include one or more base address indices 1106. Each base address index 1106 is a pointer to a base address register value representing a segment 1150 associated with network descriptor 1110. Segment 1150 includes one or more tasks to be performed on the neural network. Count parameter 1108 records the number of segments associated with the neural network. For example, count parameter 1108 counts the number of base address indices 1106 included in network descriptor 1110.

[0118] External system memory address 1112 is a pointer to an address in system memory 230 indicating a backup storage location for moving data between buffer 334 and system memory 230. Buffer 334 in data processor circuitry 318 can be a fast access memory, such as a cache memory, used to store input and output data of neural engine 314 and planar engine 340 within neural processor circuitry 218. Output data for one task may correspond to an intermediate layer in a neural network and can be used as input for another layer. Buffer 334 has a limited storage capacity and can be used to store data associated with the neural network. In some cases, neural processor circuitry 218 may perform a context switch and transition its current operation from the first neural network to the second neural network even before the computation of the first neural network is complete. In this case, the data currently associated with the first neural network stored in buffer 334 is placed at a location in system memory 230 for later retrieval when the computation associated with the first neural network is resumed. In this case, system memory 230 may be referred to as external system memory because it is located outside of neural processor circuitry 218. External memory system address 1112 records the address of the location in system memory 230 where outgoing data from buffer 334 is stored. In some embodiments, each neural network representation 1100 may have a designated location in system memory 230 to store any outgoing data from buffer 334. In those embodiments, the external system memory address 1112 for each neural network representation 1100 is unique.

[0119] Context switching refers to the process by which neural processor circuitry 218 switches from a first task (outgoing task) to a second task (incoming task) unrelated to the first task. Context switching is typically associated with a computational switch from a first neural network to a second neural network, but other types of switching (e.g., task queue switching) can also be context switching. During a context switch, kernels and data associated with the first neural network (particularly intermediate data stored in buffer 334) are swapped out to system memory 230, and data associated with the second neural network is swapped in from system memory 230.

[0120] Notification parameter 1114 and wait parameter 1116 are used to track the currently running neural network and provide an indication of a context switch. Notification parameter 1114 may take the form of allowing software (e.g., a software application or operating system) executed by CPU 208 to track one or more bit values ​​of the currently running neural network. Notification parameter 1114 may cause the neural task manager 310 to raise an interrupt when resolving the network descriptor 1110. Wait parameter 1116 may take the form of being provided to allow software to synchronously write data to system memory 230 and swap data into buffer 334 for use in an incoming task, one or more bit values. For example, wait parameter 1116 may be set to indicate that the neural processor circuitry 218 is in a context switch phase. When wait parameter 1116 is set to this phase, neural task manager 310 may pause and wait for software confirmation of a context switch.

[0121] Context switching task 1120 may take the form of a special task descriptor 1012. Context switching task 1120 specifies that data processor circuitry 318 performs a context switching process from processing outgoing tasks to processing incoming tasks. Outgoing tasks may be assigned to a first neural network, while incoming tasks may be assigned to a second neural network different from the first neural network. For example, when enqueuing context switching task 1120, neural task manager 310 may send configuration data for context switching task 1120 to data processor circuitry 318, causing data processor circuitry 318 to transfer the output of one or more neural engines 314 or planar engines corresponding to outgoing tasks from buffer 334 to system memory 230. The configuration data for context switching task 1120 may also cause data processor circuitry 318 to retrieve data corresponding to incoming tasks from system memory 230 to buffer 334.

[0122] The neural network representation 1100 also includes one or more segments 1150. Each segment 1150 is a collection of tasks associated with the neural network. Segments 1150 may take the form of task-related data and may include one or more task descriptors 1012. In some embodiments, segments 1150 may include a header 1152 and a series of task descriptors 1012.

[0123] Header 1152 stores data fields common to tasks within a specific segment 1150. Header 1152 may include an end indicator 1154, a branch indicator 1156, a first branch address 1158, a second branch address 1160, a segment mask index 1162, and a branch task identifier 1164. The end indicator 1154 indicates whether the specific segment 1150 is the last segment of the neural network representation 1100.

[0124] The neural task manager 310 supports branching, allowing it to selectively enqueue one of two or more branches from a task queue following the current segment 1150. The neural task manager 310 may receive branching commands from the data processor circuitry 318. These commands may be determined based on one or more values ​​of the output of the task in the current segment 1150. Based on the branching command, the neural task manager 310 selects one branch to enqueue. A branch indicator 1156 indicates whether branching is enabled for the current segment 1150. If so, the segment 1150 may further include a first branch address 1158 and a second branch address 1160, where the first branch address represents the address of a first subsequent segment 1150 of the first branch and the second subsequent segment 1150 of the second branch, respectively. These two branches may belong to the same neural network or different neural networks. The neural task manager 310 uses the first branch address 1158 and the second branch address 1160 to retrieve the selected segment branch for enqueueing. Branch task identifier 1162 identifies which task in segment 1150 is the branch determination task that generates output data used to determine the branch result.

[0125] Segment mask index 1162 can also be referred to as the live-in value. When a context switch occurs, segment mask index 1162 provides masking information when segment 1150 is switched out in the middle of a segment. For example, segment 1150 may include more than one task, and the context switch may occur after an intermediate task, rather than the final task, has completed. Segment mask index 1162 also provides masking information when data is swapped out of buffer 334 during a context switch.

[0126] Segment 1150 also includes one or more task descriptors 1012, which are arranged in a specific order for dequeueing by the neural task manager 310. Although the task descriptors 1012 are arranged in order, in some implementations, tasks can be executed asynchronously, such as... Figure 8 and Figure 9 As discussed in [the document]. In some implementations, such as […]. Figure 11A In the configuration shown, task descriptors 1012 may be stored within segment 1150 as part of a memory location block used to store data associated with a specific segment 1150. In other embodiments (not shown), segment 1150 may include pointers to various task descriptors 1012 stored separately from segment 1150. For example, task descriptors 1012 may be stored in system memory 230. Segment 1150 then stores a list of memory addresses representing a list of task descriptors 1012.

[0127] Figure 11BThis is a block diagram illustrating an exemplary configuration of a task queue 1004 according to one embodiment. Figure 11B Can correspond to Figure 10 A portion of the neural task manager 310 shown is illustrated, including a task arbitrator 1002 and multiple task queues 1004A to 1004N. The neural task manager 310 may include a predetermined number (e.g., eight) of individual task queues 1004. Task queues 1004 may be hardware queues including memory slots (e.g., segment slots) for storing up to a predetermined number of segments 1150. Segments 1150 from one or more neural network representations 1100 may be enqueued into and dequeued (e.g., sent for execution) from task queues 1004 based on their priority and order within the task queues 1004.

[0128] Task queue 1004 may include one or more registers for storing values ​​representing the status and configuration of task queue 1004. The registers may include a status register 1172, a free space register 1174, a queue priority register 1176, and one or more registers for the current task pointer 1178. Status register 1172 stores a value representing the execution status of the queue. Free space register 1174 indicates the number of free slots in the hardware queue. Slots may be occupied by segments 1150 and may be freed up for execution by dequeuing tasks in segment 1150. Queue priority register 1176 stores priority parameters for task queue 1004. Task arbitrator 1002 selects one of the multiple task queues 1004 to be executed based on the priority parameters. After selecting task queue 1004, segment 1150 is executed in a first-in-first-out (FIFO) manner until segment 1150 in task queue 1004 is executed or until a context switch occurs. The current task pointer 1178 provides the memory address of the current task descriptor 1012 in the first segment 1150.

[0129] In some implementations, the neural task manager 310 may support intra-queue context switching and inter-queue context switching. Intra-queue context switching occurs when a context switch exists within task queue 1004. In other words, the neural task manager 310 executes task queue 1004, and the context switch occurs between one segment and another within the same queue 1004. In inter-queue context switching, the currently executing first task queue 1004 is terminated, and the neural task manager 310 is switched to another queue.

[0130] For task queue 1004 involving in-queue context switching, segments 1150 from different neural network representations 1100 can be enqueued on the same task queue 1004, such as in an interleaved manner or in any suitable order. Execution of segments 1150 within task queue 1004 follows the order of the segments. Neural task manager 310 inserts context switching tasks 1120 between segments 1150 of different neural network representations 1100 on the same task queue 1004. Sharing the same task queue with multiple neural networks allows software and compilers to share a single priority level (with the same priority parameter) across multiple neural networks in time. When neural task manager 310 completes the first segment 1150 from the first neural network representation 1100, neural task manager 310 performs an in-queue context switch to switch to the second segment 1150 from the second neural network representation 1100. If two consecutive segments 1150 of the same neural network representation 1100 are enqueued back-to-back, no context switch is inserted.

[0131] In some cases, context switching within a queue can occur before segment 1150 is completed (e.g., tasks within the segment have not yet been dequeued). Neural Task Manager 310 uses segment mask index 1162 to store mask information for a specific segment 1150 before the context switch, allowing Neural Task Manager 310 to later return to that segment 1150.

[0132] After the neural task manager 310 completes task queue 1004, it can also perform an inter-queue context switch to dequeue another task queue 1004. A context switch can occur if two consecutive task queues 1004 contain tasks from different neural network representations 1100. For example, the last task in the first task queue 1004 can be assigned to the first neural network, while the first task in the second task queue 1004 can be assigned to the second neural network. In some cases, a context switch is not required during the transition between the two queues. For example, the first task in a subsequent task queue 1004 may simply be a continuation of the last task in the previous queue.

[0133] The value in status register 1172 can represent various execution states of task queue 1004 related to context switching. In one implementation, task queue 1004 can be in one of four states. In the enabled state, task queue 1004 participates in task arbitration and executes tasks as usual. In the stopped state, task queue 1004 participates in arbitration but does not execute tasks. If task queue 1004 becomes the current queue, neural task manager 310 performs a context switch and stops execution of any tasks in the stopped task queue 1004. In the suspended state, task queue 1004 does not participate in arbitration. At the next context switch point, arbitration will select a new queue. If no other queue exists, suspended task queue 1004 remains in the current queue. In the disabled state, task queue 1004 does not participate in arbitration. At the next context switch point, arbitration will select a new queue. If no other queue exists, disabled task queue 1004 is context-switched out to the idle state. This causes buffer 334 to overflow into system memory 230 without correspondingly filling data into buffer 334. Context switching task 1120 originates from network descriptor 1110 of the current queue from the most recent segment 1150.

[0134] Figure 12 This is a diagram illustrating a task descriptor 1012 according to one embodiment. When task queue 1004 is selected, tasks in segment 1150 are dequeued and sent for execution in sequence. By way of example, task arbitrator 1002 places task descriptor 1012 from system memory 230 into fetch queue 1008 and then transfers the task descriptor to configuration queue 1010. The highest priority (e.g., advanced) task descriptor 1012 in configuration queue 1010 is used to configure neural processor circuitry 218 for execution during a configuration cycle. Configuration data 1014 for task descriptor 1012 includes a task descriptor header 1202 and address data 1204A to 1204X (hereinafter referred to as "address data 1204").

[0135] The task descriptor header 1202 configures various operations of the neural task manager 310 related to a specific task descriptor 1012, including operations related to task selection, context switching, task switching, and data dependencies. The task descriptor header 1202 can be parsed by the task arbitrator 1002 to extract configuration data 1014 for programming other components of the neural task manager 310 and the neural processing circuitry 218.

[0136] The task descriptor header 1202 may include a task identifier (ID) 1206 that identifies the task, a task mask index 1208 that defines mask information when data is swapped out of buffer 334 during a context switch, a task switching parameter 1210 that defines whether the neural task manager 310 should initiate a task switch after the task is executed (e.g., at the end of segment 1150), an input surface parameter 1212 that defines whether input data for the task should be retrieved from system memory 230 or data processor circuitry 318, an output surface parameter 1214 that defines whether the task's output data should be stored in system memory 230 or data processor circuitry 318, various pointers (e.g., base addresses) that facilitate programming of the neural processor circuitry 218, one or more debug / exception parameters 1218 that control events, exceptions, or debug logs, and a dependency parameter 1220 that defines whether a particular task depends on a previous task.

[0137] When data is swapped out of or into buffer 334 during a context switch, task mask index 1208 provides masking information. Task mask index 1208 may also be referred to as a live-out value. During computations related to an outgoing task, not every memory location in buffer 334 needs to be occupied by data useful for computations of the outgoing task or subsequent tasks related to it. Task mask index 1208 provides masking information indicating which portion of buffer 334 needs to have data swapped in or out. For example, task mask index 1208 can be a multi-bit value, where each bit corresponds to a section in buffer 334. A bit value "1" (or vice versa) can indicate a section corresponding to a bit that needs to be swapped in or out.

[0138] Each instance of address data 1204A through 1204N (collectively or individually referred to as “address data 1204”) defines an address and data payload pair for programming components of neural processor circuitry 218. The data payload may include input data and kernel data for performing a task. For example, each instance of address data 1204 includes register data defining the data payload, a register address defining the destination memory location of neural processing circuitry 218 for receiving the register data, and a register count defining the number of consecutive memory locations (e.g., registers) to which the register data to be written. In some embodiments, the register address is combined with a base address index 1106 stored in network descriptor 1110 to define the full address of each memory location. If task descriptor 1012 is generated at compile time, the actual runtime address may be unknown. Using base address index 1106 avoids copying or updating all task descriptors with dynamically allocated addresses.

[0139] In one or more embodiments, base address index 1106 is used to program data processor circuitry 318. Base address index 1106 includes data dependency information. The data dependency information is included as part of configuration data 1014 sent to data processor circuitry 318.

[0140] Exemplary branch operations

[0141] Figure 13 This is a conceptual diagram illustrating an exemplary dynamic branching process according to one embodiment. Neural processor circuitry 218 supports the branching process, where neural task manager 310 selects one of two alternative branches to enqueue based on the result of a pre-branching task associated with a pre-branch segment. Branching allows neural processor circuitry 218 to dynamically select one or more neural networks to execute. For example, the two branches may correspond to different neural networks. Neural processor circuitry 218 selects one neural network from the neural networks based on the result of a previous neural network executed by neural processor circuitry 218.

[0142] Branching operations can be performed by a neural task manager 310 that communicates with data processor circuitry 318. Neural processor circuitry 218 can instantiate multiple neural network representations 1100. Each neural network representation 1100 may include multiple segments 1150, but for... Figure 13 Each of the three neural network representations 1100 shown illustrates only one segment 1150. The segment 1150 shown may include a pre-branch segment 1310, a first segment branch 1320, and a second segment branch 1330. Each segment may include one or more tasks described by its corresponding task descriptor 1012.

[0143] The neural task manager 310 connects the pre-branch segment 1310, the first segment branch 1320, and the second segment branch 1330 through header information in the pre-branch segment 1310. See also... Figure 11A The pre-branch segment 1310 includes a branch indicator 1156 associated with a value indicating that this segment ends with a branch. To link the branches, the pre-branch segment 1310 also includes a first branch address 1158 representing the address of the first segment branch 1320 and a second branch address 1160 representing the address of the second segment branch 1330. Although as an example... Figure 13 Two branches are shown, but in some implementations, the neural task manager 310 may support more than two branches and have corresponding branch addresses stored in the header of the pre-branch segment 1310.

[0144] Branching operations can be performed via a neural task manager 310 that communicates with data processor circuitry 318. In one embodiment, the neural task manager 310 enqueues pre-branch segments 1310 into a hardware task queue 1004. As the neural task manager 310 processes individual segments and tasks, the pre-branch segment 130 becomes the current segment, while the neural task manager 310 is dequeuing a pre-branch task 1312 for execution. When the neural task manager 310 dequeues the pre-branch task 1312 from the pre-branch segment 1310, configuration data for the task is extracted. The configuration data causes one or more computation engines 1350 to perform computations corresponding to the pre-branch task 1312 to generate output data 1352. The computation engine 1350 may be a neural engine 314 or a planar engine 340. For example, the configuration data for the pre-branch task 1312 may cause one or more neural engines 314 to perform convolution operations on the input data corresponding to the pre-branch task 1312 to generate output data 1352.

[0145] Data processor circuit 318 receives output data 1352 to generate branch command 1354. Output data 1352 may be generated from one or more neural engines 314 or planar engines 340. Data processor circuit 318 compares one or more values ​​in output data 1352 with one or more reference values ​​to generate branch command 1354, which is sent to neural task manager 310 for branching purposes. Various methods can be used to generate branch command 1354. In one embodiment, a first output value of computing engine 1350 is compared with a reference value stored in a register of data processor circuit 318. In another embodiment, another output value is selected for comparison with the reference value. For ease of comparison, the reference value is formatted the same as the output value from computing engine 1350. Data processor circuit 318 may also include registers for storing values ​​that define the manner in which the comparison is performed. For example, the comparison may be equal to, less than, less than or equal to, greater than, greater than or equal to, not equal to, always false, and always true. Based on this comparison, data processor circuit 318 generates branch command 1354 from the output data. In some implementations, the branch command 1354 may be a set of binary bits used to select one of two branches. In other implementations, the branch command 1354 may be more complex to allow selection of multiple branches.

[0146] The neural task manager 310 receives a branch command 1354 and selects a branch based on the received branch command 1354. The selected segment branch is enqueued into a task queue 1004. In one embodiment, the enqueued task queue 1004 is the same task queue to which the pre-branched segment 1310 is enqueued. In another embodiment, the selected segment branch may also be enqueued into a different task queue 1004 with different priority parameters. The neural task manager 310 enqueues the selected segment branch by retrieving the branch based on a first branch address 1158 or a second branch address 1160 specified in the segment header 1152. The neural task manager 310 then transmits the task from the selected branch in the branch to the data processor circuitry 318 for execution of the task. For example, configuration data for the task is extracted and used to guide the data processor circuitry 318 in retrieving appropriate data for the computation engine 1350 to perform computations.

[0147] Although the first segment branch 1320, the second segment branch 1330, and the pre-branch segment 1310 are in Figure 13 The segments 1310 are represented as being assigned to different neural network representations 1100, but in some implementations, one or more segments of those segments 1310 may also belong to the same neural network representation 1100. If the pre-branch segments 1310 and the selected segment branches are assigned to different neural network representations 1100, the neural task manager 1310 may also include a context switching task 1120 (see Network Descriptor 1110 containing the neural network representation 1100 of the pre-branch segments 1310) stored in the network descriptor 1110. Figure 11A Enqueue. In some cases, the last task of the pre-branch segment 1310 is the last task associated with the neural network, in which case context switching may not be required.

[0148] In various implementations, which pre-branch task 1312 determines the branching result may differ. In some implementations, the pre-branch task 1312 that determines the branching result is the last task in the pre-branch segment 1310. The branch command 1354 causes the selected segment branch to be added to the same task queue 1004 currently storing the pre-branch segment 1310. Since the neural task manager 310 is completing the last task in the pre-branch segment 1310 in this example, the task queue 1004 will have at least one segment slot for accepting the selected segment branch. In one implementation, if the neural task manager 310 has already dequeued the last pre-branch task 1312 for execution but has not yet received the branch command 1354 (e.g., in the case that the last pre-branch task 1312 is a long and complex task), the neural task manager 310 may pause and stop dequeuing any additional tasks to be executed.

[0149] In some implementations, the neural task manager 310 allows any pre-branch task 1312 in the pre-branch segment 1310 to be used as a branch-determining task, rather than fixing the last pre-branch task 1312 as the branch-determining task. The header 1152 of the pre-branch segment 1310 includes a branch task identifier 1162 that identifies the branch-determining task. For example, in... Figure 13 In this context, the third pre-branch task 1312 is identified as the task that will produce output data 1352 for generating the branch command 1354. When an intermediate task is selected as the branch determination task, the neural task manager 310 continues processing the additional pre-branch tasks 1312 before the branch occurs. For example, two additional pre-branch tasks 1312 are dequeued before the branch occurs.

[0150] In some implementations, the neural task manager 310 supports continuing task processing after the branch determination task has been dequeued and before the branch command 1354 is received. For example, the branch determination task can be a complex task that takes a relatively long time to process (e.g., Figure 8 In task TC3), neural task manager 130 can continue processing subsequent pre-branch tasks 1312 in pre-branch segment 1310. If all pre-branch tasks 1312 have been dequeued but have not yet received branch command 1354, neural task manager 130 can enqueue additional tasks. Any suitable task can be enqueued, including context switching tasks and those subsequent tasks that are related to the branch determination task. The continued operation of neural task manager 310 while waiting for branch results improves the efficiency of neural processor circuitry 218. While neural task manager 310 waits for branch results, it can process additional tasks, such as those related to another neural network.

[0151] In some implementations, the neural task manager 310 is a segment 1150 with branches capable of context switching. The neural task manager 310 monitors the branch status of the foremost segment 1150 in each task queue 1004. If a branch determines that a task has been completed, the branch status can be the branch command 1354 for that task. For example, if in Figure 13If a context switch occurs at the fourth pre-branch task 1312 and the third pre-branch task 1312 is a branch-determined task, then the neural task manager 310 stores the branch command 1354 (if available) as a branch state. If the branch-determined task is not yet complete, the branch state can also be the branch task identifier 1162. This occurs when the neural task manager 310 is still waiting for the branch command 1354 or when the context switch occurs before the branch-determined task is dequeued. If the neural task manager 310 reaches the end of the currently executing segment and the branch command 1354 is unavailable, the neural task manager 310 can also perform a context switch to another task queue 1004 to handle other tasks.

[0152] In some implementations, the neural task manager 310 may also begin processing one task within a segmented branch (first segmented branch 1320 or second segmented branch 1330) before receiving the branch command 1354. For example, in some cases, the neural task manager 310 may predict or randomly select a branch, enqueue the predicted branch, and execute operations related to the predicted branch before receiving the branch command 1354. If the predicted branch result is the selected branch specified by the branch command 1354, the neural task manager 310 has accelerated the computation process. If the predicted branch result is incorrect, the neural task manager 310 discards operations related to the predicted branch and clears the predicted branch. The neural task manager 310 then enqueues the correct branch.

[0153] Figure 13 The branches shown can be assigned to different neural networks. For example, a branch can correspond to an operation of a set of machine learning models, where the selection of two alternative models is based on the result of the previous model. For example, the pre-branch segment 1310 can correspond to a first classifier. After determining the result of the first classifier, the neural processor circuit 218 can immediately switch to one of the two alternative subsequent classifiers for more refined predictions. Using the neural processor circuit 218 for branch selection improves the latency of performing operations related to a series of machine learning models. For example, the decision of the first classifier does not need to be transmitted back to the application layer for the software to decide which subsequent classifier to choose.

[0154] Exemplary process for branching

[0155] Figure 14This is a flowchart depicting an exemplary process for performing a branching operation in a neural processor circuit 218 according to one embodiment. This process can be performed collaboratively by various components of the neural processor circuit 218. For example, branching can occur when the neural processor circuit 218 determines a result associated with a first neural network and transitions to performing computation using a second neural network selected from two or more choices of the first neural network.

[0156] Neural processor circuit 218 performs a convolution operation 1410 on input data corresponding to a pre-branching task via one or more neural engine circuits 314 to generate output data. The convolution operation may correspond to operations in one or more convolutional layers in a CNN. The convolution operation may also correspond to operations in other types of machine learning models. In some cases, the pre-branching task used to generate output data that leads to a branching command may also be performed by planar engine 340.

[0157] Neural processor circuit 218 receives output data 1420 from one or more neural engine circuits via data processor circuit 318. In some embodiments, the output data may be the result of convolution. Data processor circuit 318 may use the entire output data to determine the branch. In other embodiments, data processor circuit 318 may select one value from the output data or sample a subset of the values ​​to determine the branch. For example, in one embodiment, neural processor circuit 318 may use a first value from the output data as the value used to determine the branch.

[0158] Neural processor circuit 218 generates 1430 branch commands from the output data. For example, branch commands can be generated based on comparing one or more values ​​in the output data with one or more reference values. In one embodiment, data processor circuit 318 includes a register storing reference values. The output value of the output data is compared with the reference values ​​to generate a branch command, which can be a binary value representing two possibilities for a binary branch. If more than two branches are involved, the branch command can also be more complex.

[0159] Neural processor circuit 218 receives branch commands 1440 from data processor circuitry via neural task manager circuitry 310. In some embodiments, if neural task manager 310 has not yet received a branch command and neural task manager 310 determines that tasks in a pre-branched task segment have been transmitted for execution, neural task manager 310 may be paused until a branch command is received. In other embodiments, neural task manager 130 may process individual task segments different from two or more segmented branches before receiving a branch command.

[0160] The neural task manager 310 enqueues two or more segmented branches 1450 based on received branch commands, these two or more segmented branches following a pre-branch task segment that includes the pre-branch task. The neural task manager 310 can identify two or more segmented branches by the addresses of those segmented branches stored in the pre-branch task segment. In some embodiments, the selected segmented branch is enqueued into the task queue in which the pre-branch task segment is enqueued.

[0161] Data processor circuit 318 transfers tasks from selected segment branches within segmented branches to the data processor circuit for execution. For example, it extracts configuration data for the task from a task descriptor. The configuration data causes data processor circuit 318 to retrieve data from buffer 334 and transfer the data to neural engine 314 or planar engine 340 for computation. Tasks in the pre-branched task segments can be assigned to a first neural network, and tasks in selected segment branches can be assigned to a second neural network different from the first neural network. Therefore, neural processor circuit 218 selects subsequent neural networks based on the results of the previous neural network.

[0162] In some implementations, after the pre-branch task for determining the branch is dequeued for execution, the data processor circuitry 318 may spend a relatively long time receiving the output data of the branch determination task to generate a branch command. For example, the branch determination task may be a long task to be completed. In some cases, the remaining tasks in the pre-branch have been dequeued while waiting for the branch command. In various implementations, the neural task manager 310 may have different methods for handling such situations. In one implementation, the neural task manager 310 may pause until a task command is received. In another implementation, the neural task manager 310 may perform other tasks unrelated to the branch. For example, the neural task manager 310 may perform a context switch to utilize the waiting time to perform other neural network operations unrelated to the branch. In yet another implementation, the neural task manager 310 may make a prediction or randomly select a branch to dequeue before receiving the branch command. If the prediction is incorrect, the neural task manager 310 switches to the correct branch and discards the work done for the incorrectly predicted task.

[0163] While specific implementations and applications have been described and illustrated, it should be understood that the invention is not limited to the precise constructions and components disclosed herein, and that various modifications, alterations, and variations that will be apparent to those skilled in the art may be made to the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. A neural processor circuit, comprising: One or more neural engine circuits, the one or more neural engine circuits being configured to perform convolution operations on input data corresponding to a pre-branch task to generate output data; A data processor circuit, coupled to the one or more neural engine circuits, is configured to: Receive the output data from the one or more neural engine circuits; as well as Generate branch commands from the output data; and Task manager circuitry, coupled to the data processor circuitry, is configured to: Receive the branch command from the data processor circuit; Based on the received branch command, enqueue one of two or more segmented branches, which are after the pre-branch task segment that includes the pre-branch task; as well as The task from one of the two or more segmented branches is transferred to the data processor circuit to execute the task.

2. The neural processor circuit of claim 1, wherein the pre-branch task segment is assigned to a first neural network, and one selected segment branch of the two or more segment branches is assigned to a second neural network different from the first neural network.

3. The neural processor circuit of claim 1, wherein the branch command is generated based on comparing one or more values ​​in the output data with one or more reference values.

4. The neural processor circuit of claim 1, wherein a selected segment branch of the two or more segment branches is enqueued into the task queue in which the pre-branch task segment is enqueued.

5. The neural processor circuit of claim 1, wherein the addresses of the two or more segmented branches are stored in the pre-branched task segment.

6. The neural processor circuit of claim 1, wherein the pre-branch task for determining the branch command is identified by a branch task identifier stored in the pre-branch task segment.

7. The neural processor circuit of claim 1, wherein the pre-branch task that determines the branch command is the last task in the pre-branch task segment.

8. The neural processor circuit of claim 1, wherein the pre-branch task that determines the branch command is an intermediate task in the pre-branch task segment.

9. The neural processor circuit of claim 1, wherein the task manager circuit is further configured to: Before receiving the branch command, it is determined that the tasks in the pre-branch task segment have been transmitted for execution; and Pause the Task Manager circuit until the branch command is received.

10. The neural processor circuit of claim 1, wherein the task manager circuit is further configured to: Before receiving the branch command, it is determined that the tasks in the pre-branch task segment have been transmitted for execution; and Before receiving the branch command, process individual task segments that are different from the two or more segment branches.

11. A method for performing neural processing operations, comprising: One or more neural engine circuits perform convolution operations on the input data corresponding to the pre-branch task to generate output data; The output data is received from the one or more neural engine circuits by the data processor circuit; Generate branch commands from the output data; The branch command is received by the task manager circuit from the data processor circuit; Based on the received branch command, enqueue one of two or more segmented branches, which are after the pre-branch task segment that includes the pre-branch task; as well as The task from one of the two or more segmented branches is transferred to the data processor circuit to execute the task.

12. The method of claim 11, wherein the pre-branch task segment is assigned to a first neural network, and one selected segment branch of the two or more segment branches is assigned to a second neural network different from the first neural network.

13. The method of claim 11, wherein the branch command is generated based on comparing one or more values ​​in the output data with one or more reference values.

14. The method of claim 11, wherein a selected segment branch of the two or more segment branches is enqueued into the task queue in which the pre-branch task segment is enqueued.

15. The method of claim 11, wherein the pre-branch task of determining the branch command is an intermediate task in the pre-branch task segment.

16. The method of claim 11, further comprising: Before receiving the branch command, it is determined that the tasks in the pre-branch task segment have been transmitted for execution; as well as Before receiving the branch command, process individual task segments that are different from the two or more segment branches.

17. An electronic device comprising: System memory, wherein the system memory stores one or more machine learning models; and A neural processor, the neural processor comprising: One or more neural engine circuits, the one or more neural engine circuits being configured to perform convolution operations on input data corresponding to a pre-branch task to generate output data; A data processor circuit, coupled to the one or more neural engine circuits, is configured to: Receive the output data from the one or more neural engine circuits; and Generate branch commands from the output data; and Task manager circuitry, coupled to the data processor circuitry, is configured to: Receive the branch command from the data processor circuit; Based on the received branch command, enqueue one of two or more segmented branches, said two or more segmented branches following a pre-branch task segment that includes the pre-branch task; and The task from one of the two or more segmented branches is transferred to the data processor circuit to execute the task.

18. The electronic device of claim 17, wherein the pre-branch task segment is assigned to a first neural network, and one selected segment branch of the two or more segment branches is assigned to a second neural network different from the first neural network.

19. The electronic device of claim 17, wherein the branch command is generated based on comparing one or more values ​​in the output data with one or more reference values.

20. The electronic device of claim 17, wherein the branch command corresponds to a prediction result of one of the one or more machine learning models.

Images