Multiple operation modes of a neural engine circuit

Abstract

Implementations relate to a neural engine circuit of a neural network processor circuit that performs a convolution operation on input data in a first mode and performs a parallel sort operation on input data in a second mode. The neural engine circuit includes a plurality of operation circuits and an accumulator circuit coupled to the plurality of operation circuits. The plurality of operation circuits receive input data. In the first mode, the plurality of operation circuits perform multiply-accumulate operations of a convolution on the input data using a kernel. In the second mode, the plurality of operation circuits perform a portion of a parallel sort operation on the input data. In the first mode, the accumulator circuit receives and stores first results of the multiply-accumulate operations. In the second mode, the accumulator circuit receives and stores second results of the parallel sort operation.

Description

Background Technology 1. Technical Field

[0002] This disclosure relates to a circuit for performing operations related to neural networks, and more specifically, to a neural engine circuit that performs convolution operations in one mode and parallel sorting operations in another mode.

[0003] 2. Relevant Technical Descriptions

[0004] Artificial neural networks (ANNs) are computational systems or models that use a set of connected nodes to process input data. ANNs are typically organized into layers, with different layers performing different types of transformations on their inputs. Extensions or variants of ANNs such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs) have received considerable attention. These computational systems or models typically involve a wide range of computational operations, including multiplication and accumulation. For example, CNNs are a class of machine learning techniques that primarily use convolutions between input data and kernel data; convolutions can be decomposed into multiplication and accumulation operations.

[0005] These machine learning systems or models can be configured differently depending on the type of input data and the operations to be performed. Such varied configurations would include, for example, preprocessing operations, the number of channels in the input data, the kernel data to be used, the nonlinear functions to be applied to the convolution results, and the application of various post-processing operations. Instantiating and executing machine learning systems or models with various configurations using a central processing unit (CPU) and its main memory is relatively straightforward, as such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for the various operations of these machine learning systems or models would consume significant CPU bandwidth and increase overall power consumption. Summary of the Invention

[0006] The embodiment relates to a neural engine circuit of a neural network processor circuit, which performs convolution operations on input data in a first mode and performs parallel sorting operations on the input data in a second mode. The neural engine circuit includes multiple operational circuits and an accumulator circuit coupled to the outputs of the multiple operational circuits. The operational circuits receive input data. In the first mode, the multiple operational circuits perform a convolution multiplication-addition operation on the input data using kernels. In the second mode, the multiple operational circuits perform a portion of a parallel sorting operation on the input data. In the first mode, the accumulator circuit receives and stores a first result of the multiplication-addition operation. In the second mode, the accumulator circuit receives and stores a second result of the parallel sorting operation.

[0007] In one or more embodiments, in this second mode, the second result of the parallel sorting operation is sent to the plurality of arithmetic circuits to execute subsequent parts of the parallel sorting operation.

[0008] In one or more implementations, the parallel sorting operation implements a bitonic sorting network that simultaneously generates the maximum and minimum values ​​of the input data.

[0009] In one or more embodiments, the accumulator circuit receives and stores in the second mode an index for the maximum value of the input data and an index for the minimum value of the input data.

[0010] In one or more embodiments, each arithmetic circuit in the arithmetic circuitry includes a multiplier circuit, an adder circuit, and a comparator circuit. In a first mode, the multiplier circuit receives a portion of the input data and a portion of the core, and performs a multiplication operation on that portion of the input data and that portion of the core. The adder circuit is coupled to the multiplier circuit and the accumulator circuit. In the first mode, the adder circuit receives a stored first result and a value derived from the multiplication operation, and performs an addition operation on the stored first result and the value derived from the multiplication operation. The comparator circuit is coupled to the accumulator circuit. In a second mode, the comparator circuit receives a portion of the input data and a stored second result, and performs a comparison operation on that portion of the input data and the stored second result.

[0011] In one or more embodiments, the neural engine circuitry further includes an input buffer circuitry coupled to the computational circuitry to provide different sets of input data to the plurality of computational circuitry in different cycles of the neural engine circuitry.

[0012] In one or more embodiments, another neural engine circuit performs another portion of the parallel sorting operation in parallel with the neural engine circuit.

[0013] In one or more embodiments, the input data includes a plurality of input data elements, and the plurality of arithmetic circuits are further configured to perform a portion of the parallel sorting operation on the plurality of input data elements in the second mode. Attached Figure Description

[0014] Figure 1 It is a high-level diagram of an electronic device according to an implementation plan.

[0015] Figure 2 This is a block diagram illustrating components in an electronic device according to one embodiment.

[0016] Figure 3 This is a block diagram illustrating a neural processor circuit according to one embodiment.

[0017] Figure 4A It is a block diagram of a neural engine based on an implementation scheme.

[0018] Figure 4B This is a block diagram illustrating the operation of a neural engine in a second mode according to one implementation scheme.

[0019] Figure 5 This is a circuit diagram of the computational circuit and accumulator circuit of a neural engine according to one implementation scheme.

[0020] Figure 6 This is a flowchart illustrating a method for operating a neural engine circuit in a first mode and a second mode, according to an embodiment.

[0021] Figure 7 This is a conceptual diagram illustrating an exemplary parallel sorting network executed at a neural processor circuit according to one embodiment.

[0022] For illustrative purposes only, the accompanying drawings and detailed descriptions depict various non-limiting embodiments. Detailed Implementation

[0023] Reference will now be made in detail to the embodiments, examples of which are shown in the accompanying drawings. Numerous specific details are shown in the following detailed description to provide a full understanding of the various described embodiments. However, the embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the various aspects of the embodiments.

[0024] This disclosure relates to a neural engine circuit of a neural processor circuit that performs a convolution operation on input data in a first mode and a parallel sorting operation on the input data in a second mode. The neural engine circuit includes multiple arithmetic circuits that operate in conjunction with an accumulator. During the first mode, the arithmetic circuits perform a multiply-add operation of convolution on the input data and kernel data. During the second mode, the arithmetic circuits perform a portion of the parallel sorting operation on the input data.

[0025] Exemplary electronic devices

[0026] This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functionalities such as a personal digital assistant (PDA) and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Devices, Apple Equipment and Device. Alternatively, other portable electronic devices, such as wearable devices, laptops, or tablets, may be used. In some embodiments, the device is not a portable communication device, but a desktop computer or other computing device not designed for portable use. In some embodiments, the disclosed electronic device may include a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). The following is combined with… Figure 1 The described example electronic device (e.g., device 100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0027] Figure 1 This is a high-level diagram of an electronic device 100 according to one embodiment. Device 100 may include one or more physical buttons, such as a "home" button or a menu button 104. Menu button 104 is used, for example, to navigate to any application in a set of applications running on device 100. In some embodiments, menu button 104 includes a fingerprint sensor for recognizing a fingerprint on menu button 104. The fingerprint sensor can be used to determine whether the finger on menu button 104 has a fingerprint that matches a fingerprint stored for unlocking device 100. Alternatively, in some embodiments, menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touchscreen.

[0028] In some embodiments, device 100 includes a touchscreen 150, a menu button 104, a push-button 106 for powering the device on / off and for locking the device, a volume control button 108, a subscriber identity module (SIM) card slot 110, a headset jack 112, and a docking / charging external port 124. The push-button 106 can be used to power the device on / off by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 includes various components, including but not limited to memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include more than one type of image sensor 164. Each type may include more than one image sensor 164. For example, one type of image sensor 164 may be a camera, and another type of image sensor 164 may be an infrared sensor for facial recognition performed by one or more machine learning models stored in device 100. Device 100 may include Figure 1 Components not shown include, for example, an ambient light sensor, a dot projector, and a floodlight illuminator for supporting facial recognition.

[0029] Device 100 is merely one example of an electronic device, and device 100 may have more or fewer components than those listed above, some of which may be combined into a single component or have different configurations or arrangements. The various components of device 100 listed above are embodied in hardware, software, firmware, or combinations thereof, including one or more signal processing and / or application-specific integrated circuits (ASICs).

[0030] Figure 2 This is a block diagram illustrating components in device 100 according to one embodiment. Device 100 is capable of performing various operations, including implementing one or more machine learning models. For this and other purposes, device 100 may include an image sensor 202, a system-on-a-chip (SOC) component 204, system memory 230, permanent memory (e.g., flash memory) 228, a motion sensor 234, and a display 216, among other components. Figure 2 The components shown are merely illustrative. For example, device 100 may include... Figure 2Other components not shown (such as speakers or microphones). Additionally, some components (such as motion sensor 234) may be omitted from device 100.

[0031] Image sensor 202 is a component for capturing image data and may be embodied as, for example, a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, camera, camcorder, or other device. Image sensor 202 generates raw image data, which is sent to SOC component 204 for further processing. In some embodiments, the image data processed by SOC component 204 is displayed on display 216, stored in system memory 230, permanent memory 228, or transmitted to a remote computing device via a network connection. The raw image data generated by image sensor 202 may be a Bayer color kernel array (CFA) pattern.

[0032] Motion sensor 234 is a component or set of components used to sense the motion of device 100. Motion sensor 234 can generate sensor signals indicating the orientation and / or acceleration of device 100. The sensor signals are sent to SOC component 204 for various operations, such as turning on device 100 or rotating an image displayed on display 216.

[0033] Display 216 is a component for displaying images generated by SOC component 204. Display 216 may include, for example, a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) device. Based on data received from SOC component 204, display 216 may display various images, such as menus, selected operating parameters, images captured by image sensor 202 and processed by SOC component 204, and / or other information (not shown) received from the user interface of device 100.

[0034] System memory 230 is a component used to store instructions executed by SOC component 204 and to store data processed by SOC component 204. System memory 230 can be embodied in any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or combinations thereof.

[0035] Permanent memory 228 is a component used to store data in a non-volatile manner. Even when power is unavailable, permanent memory 228 retains the data. Permanent memory 228 may be embodied as read-only memory (ROM), flash memory, or other non-volatile random access memory devices. Permanent memory 228 stores the operating system and various software applications of device 100. Permanent memory 228 may also store one or more machine learning models, such as regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional networks (CNNs), recurrent networks (RNNs), autoencoders, and long short-term memory (LSTMs). Machine learning models can be standalone models that work with neural processor circuitry 218 and various software applications or sensors of device 100. Machine learning models can also be part of software applications. Machine learning models can perform various tasks, such as face recognition, image classification, object, concept and information classification, speech recognition, machine translation, voice recognition, voice command recognition, text recognition, text and context analysis, other natural language processing, prediction, and suggestion.

[0036] Various machine learning models stored in device 100 can be fully trained, untrained, or partially trained to allow device 100 to enhance or continue training the machine learning models as device 100 is used. The operation of the machine learning models includes various computations, such as training the model and using the model to determine runtime results. For example, in one scenario, device 100 captures a user's facial image and uses that image to further improve the machine learning model used to lock or unlock device 100.

[0037] SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing procedures. SOC component 204 may include an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processing unit (GPU) 220, a memory controller 222, a video encoder 224, a memory controller 226, and a bus 232 connecting these sub-components, as well as other sub-components. SOC component 204 may include... Figure 2 The sub-components shown have more or fewer sub-components.

[0038] ISP 206 is a circuit that performs each stage of the image processing pipeline. In some implementations, ISP 206 may receive raw image data from image sensor 202 and process the raw image data into a form available to other sub-components of SOC component 204 or components of device 100. ISP 206 may perform various image manipulation operations, such as image panning, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations.

[0039] CPU 208 can be implemented using any suitable instruction set architecture and can be configured to execute instructions defined in that instruction set architecture. CPU 208 can be a general-purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, RISC, ARM, or MIPS ISA, or any other suitable ISA. Although Figure 2 A single CPU is shown, but SOC component 204 may include multiple CPUs. In a multiprocessor system, each CPU may collectively implement the same ISA, but this is not required.

[0040] The graphics processing unit (GPU) 220 is a graphics processing circuit for executing graphics data. For example, the GPU 220 may render objects to be displayed in a frame buffer (e.g., a frame buffer that includes pixel data for the entire frame). The GPU 220 may include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of some graphics operations.

[0041] Neural processor circuit 218 is a circuit that performs various machine learning operations based on computations including multiplication, addition, comparison, and accumulation. Such computations can be arranged, for example, to perform various types of tensor multiplication, such as tensor products and convolutions of input data and kernel data. Neural processor circuit 218 is a configurable circuit that performs these operations in a fast and efficient manner, while alleviating the resource-intensive operations associated with neural network operations on CPU 208. Neural processor circuit 218 can receive input data from sensor interface 212, image signal processor 206, permanent memory 228, system memory 230, or other sources such as network interface 210 or GPU 220. The output of neural processor circuit 218 can be provided to various components of device 100, such as image signal processor 206, system memory 230, or CPU 208, for various operations. The structure and operation of neural processor circuit 218 are referenced below. Figure 3 Detailed description.

[0042] Network interface 210 is a sub-component that supports the exchange of data between device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data may be received from other devices via network interface 210 and stored in system memory 230 for subsequent processing (e.g., via a back-end interface to image signal processor 206) and display. Networks may include, but are not limited to, local area networks (LANs) (e.g., Ethernet or corporate networks) and wide area networks (WANs). Image data received via network interface 210 may be processed by ISP 206.

[0043] Sensor interface 212 is a circuit used to communicate with motion sensor 234. Sensor interface 212 receives sensor information from motion sensor 234 and processes the sensor information to determine the orientation or movement of device 100.

[0044] Display controller 214 is a circuit used to send image data to be displayed on display 216. Display controller 214 receives image data from ISP 206, CPU 208, graphics processor or system memory 230, and processes the image data into a format suitable for display on display 216.

[0045] The memory controller 222 is circuitry for communicating with the system memory 230. The memory controller 222 can read data from the system memory 230 for processing by the ISP 206, CPU 208, GPU 220, or other sub-components of the SOC component 204. The memory controller 222 can also write data to the system memory 230 received from various sub-components of the SOC component 204.

[0046] The video encoder 224 is hardware, software, firmware, or a combination thereof, used to encode video data into a format suitable for storage in permanent memory 228, or to pass data to network interface 210 for transmission over a network to another device.

[0047] In some implementations, one or more sub-components of SOC component 204, or some functions of these sub-components, may be executed by software components that run on neural processor circuitry 218, ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, permanent memory 228, or in another device that communicates with device 100 via network interface 210.

[0048] Example Neural Processor Circuit

[0049] The neural processor circuit 218 is a programmable circuit that performs machine learning operations on the input data of the neural processor circuit 218. The machine learning operations may include different computations for training a machine learning model and for performing inference or prediction based on the trained machine learning model.

[0050] Taking a CNN as an example of a machine learning model, CNN training can include forward propagation and backward propagation. A neural network can include an input layer, an output layer, and one or more intermediate layers, which may be called hidden layers. Each layer can include one or more nodes, which may be fully or partially connected to other nodes in adjacent layers. In forward propagation, the neural network performs computations in the forward direction based on the output of the previous layer. The operations of a node can be defined by one or more functions. Functions defining node operations can include various computational operations, such as data convolution using one or more kernels, parallel data sorting, layer pooling, tensor multiplication, etc. Functions can also include activation functions that adjust the weights of the node's output. Nodes in different layers can be associated with different functions. For example, a CNN may include one or more convolutional layers mixed with pooling layers and followed by one or more fully connected layers.

[0051] Each function (including the kernel) in a machine learning model can be associated with different coefficients that can be adjusted during training. Additionally, individual nodes in a neural network can also be associated with activation functions that determine the weights of the node's output during forward propagation. Common activation functions include step functions, linear functions, sigmoid functions, hyperbolic tangent functions (tanh), and rectified linear unit functions (ReLU). After a batch of training sample data passes through the neural network in forward propagation, the result is compared to the training labels of the training samples to calculate the network's loss function, which represents the network's performance. Furthermore, the neural network performs backpropagation by adjusting the coefficients in the individual functions using coordinate descent techniques such as stochastic coordinate descent (SGD) to improve the value of the loss function.

[0052] During training, device 100 may use neural processor circuitry 218 to perform all or some of the operations in forward and backward propagation. Multiple rounds of forward and backward propagation may be performed by neural processor circuitry 218 alone or in coordination with other processors such as CPU 208, GPU 220, and ISP 206. Training may be completed when the loss function no longer improves (e.g., the machine learning model has converged) or after a predetermined number of rounds for a specific training sample. While using device 100, it may continue to collect additional training samples for the neural network.

[0053] To make predictions or inferences, device 100 may receive one or more input samples. Neural processor circuitry 218 may take the input samples and perform forward propagation to determine one or more results. Input samples may be images, speech, text files, sensor data, or other data.

[0054] In machine learning, data and functions (e.g., input data, kernels, functions, layer outputs, gradient data) can be stored and represented by one or more tensors. Common operations related to the training and runtime of machine learning models can include tensor product, tensor transpose, element-wise tensor operations, convolution, sorting, application of activation functions, automatic differentiation to determine gradients of values ​​in tensors, statistics and aggregations (e.g., mean, variance, standard deviation), tensor rank and size manipulation, etc.

[0055] While the training and runtime of neural networks have been discussed as examples, the neural processor circuit 218 can also be used to operate other types of machine learning models, such as kernel SVMs.

[0056] See Figure 3 The exemplary neural processor circuitry 218 may include, among other components, a neural task manager 310, a plurality of neural engines 314A to 314N (collectively referred to below as "a plurality of neural engines 314" and also individually as "neural engine 314"), kernel direct memory access (DMA) 324, data processor circuitry 318, data processor DMA 320, and planar engine 340. The neural processor circuitry 218 may include fewer components or Figure 3 Additional components not shown.

[0057] Each neural engine in neural engine 314 performs machine learning computational operations in parallel. Depending on the workload, the entire set of neural engines 314 may be operating, or only a subset of neural engines 314 may be operating, while the remaining neural engines 314 are placed in a power-saving mode to conserve power. Each neural engine in neural engine 314 includes components for storing one or more kernels, for performing multiplication-accumulation operations, for performing parallel sorting operations, and for post-processing to generate output data 328, as described below. Figure 4A and Figure 4B Detailed description. The Neural Engine 314 is specifically designed to perform computationally intensive operations, such as convolution and tensor product operations. Convolution operations can include different types of convolution, such as cross-channel convolution (convolution that sums values ​​from different channels), communication-by-communication convolution, and transposed convolution.

[0058] Planar engine 340 can specialize in performing simpler computational operations, and its speed may depend primarily on the input and output (I / O) speed of data transfer rather than the computational speed within planar engine 340. These computational operations may be referred to as I / O-constrained computations. In contrast, neural engine 314 can focus on complex computations, and its speed may depend primarily on the computational speed within each neural engine 314. For example, planar engine 340 is efficient for performing operations within a single channel, while neural engine 314 is efficient for performing operations across multiple channels that may involve heavy data accumulation. Computations using neural engine 314 to perform I / O-constrained computations cannot be efficient in both speed and power consumption. In one implementation, the input data may be a tensor with a rank greater than three (e.g., having three or more dimensions). One set of dimensions (two or more) in the tensor may be referred to as a plane, and another dimension as a channel. Neural engine 314 may utilize kernels to convolve the plane data in the tensor and accumulate the results of different plane convolutions across different channels. On the other hand, planar engine 340 can specialize in in-plane operations.

[0059] The circuitry of the planar engine 340 can be programmed to operate in one of several modes, including pooling mode, element-wise mode, and shrinking mode. In pooling mode, the planar engine 340 shrinks the spatial size of the input data. In element-wise mode, the planar engine 340 generates an output derived from element-wise operations on one or more inputs. In shrinking mode, the planar engine 340 shrinks the rank of a tensor. For example, a rank 5 tensor can be shrunk to a rank 2 tensor, or a rank 3 tensor can be shrunk to a rank 0 tensor (e.g., a scalar).

[0060] The neural task manager 310 manages the overall operation of the neural processor circuitry 218. The neural task manager 310 may receive a list of tasks from a compiler executed by the CPU 208, store tasks in its task queue, select tasks to be executed, and send task commands to other components of the neural processor circuitry 218 for executing the selected tasks. Data may be associated with task commands indicating the type of operation to be performed on the data. The data of the neural processor circuitry 218 includes input data transferred from another source, such as system memory 230, and data generated by the neural processor circuitry 218 in previous operating cycles. Each dataset may be associated with a task command specifying the type of operation to be performed on the data. The neural task manager 310 may also perform task switching upon detecting events such as receiving instructions from the CPU 208. In one or more embodiments, the neural task manager 310 sends raster information to components of the neural processor circuitry 218 to enable each of these components to track, retrieve, or process appropriate portions of the input data and kernel data. For example, the neural task manager 310 may include registers storing information about the size and rank of the datasets for processing by the neural processor circuitry 218. Although in Figure 3 The neural task manager 310 is shown as part of the neural processor circuitry 218, but the neural task manager 310 may be a component external to the neural processor circuitry 218.

[0061] Kernel DMA 324 is a read circuit that retrieves kernel data from a source (e.g., system memory 230) and sends kernel data 326A to 326N to each neural engine in neural engine 314. Kernel data represents information from which kernel elements can be extracted. In one embodiment, the kernel data may be a compressed format that is decompressed at each neural engine in neural engine 314. Although in some cases the kernel data provided to each neural engine in neural engine 314 may be the same, in most cases the kernel data provided to each neural engine in neural engine 314 is different. In one embodiment, the direct memory access nature of kernel DMA 324 allows kernel DMA 324 to directly retrieve and write data from the source without the involvement of CPU 208.

[0062] Data processor circuitry 318 manages the data flow and task performance of neural processor circuitry 218. Data processor circuitry 318 may include flow control circuitry 332 and buffer 334. Buffer 334 is a temporary storage device for storing data associated with the operation of neural processor circuitry 218 and planar engine 340, such as input data transferred from system memory 230 (e.g., data from a machine learning model) and other data generated within neural processor circuitry 218 or planar engine 340. The data stored in data processor circuitry 318 may include different subsets, which are sent to various downstream components, such as neural engine 314 and planar engine 340.

[0063] In one embodiment, buffer 334 is implemented as a non-transitory memory accessible to neural engine 314 and planar engine 340. Buffer 334 may store input data 322A to 322N for feeding to corresponding neural engines 314A to 314N or planar engine 340, and output data 328A to 328N from each of neural engines 314A to 314N or planar engine 340 for feedback to one or more neural engines 314 or planar engine 340, or for transmission to target circuitry (e.g., system memory 230). Buffer 334 may also store input data 342 and output data 344 of planar engine 340, and allows data exchange between neural engine 314 and planar engine 340. For example, one or more output data 328A to 328N of neural engine 314 may be used as input 342 to planar engine 340. Similarly, the output 344 of planar engine 340 may be used as input data 322A to 322N of neural engine 314. The input to the neural engine 314 or the planar engine 340 can be any data stored in the buffer 334. For example, in each operation loop, the source dataset from which one of the engines draws input can be different. The input to an engine can be the output of the same engine in a previous loop, the output of a different engine, or any other suitable source dataset stored in the buffer 334. Furthermore, the dataset in the buffer 334 can be partitioned and sent to different engines for different operations in the next operation loop. Two datasets in the buffer 334 can also be combined for the next operation.

[0064] The flow control circuit 332 of the data processor circuit 318 controls the data exchange between the neural engine 314 and the planar engine 340. The operation of the data processor circuit 318 and other components of the neural processor circuit 218 is coordinated so that input data and intermediate data stored in the data processor circuit 318 can be reused across multiple operations at the neural engine 314 and the planar engine 340, thereby reducing the data transfer to and from the system memory 230. The flow control circuit 332 may perform one or more of the following operations: (i) monitor the size and rank of the data being processed by the neural engine 314 and the planar engine 340 (e.g., the data may be one or more tensors), (ii) determine which subsets of data are transmitted to the neural engine 314 or the planar engine 340 based on task commands associated with different subsets of data, (iii) determine the manner in which data is transmitted to the neural engine 314 and the planar engine 340 (e.g., the data processor circuit 318 may operate in a broadcast mode, in which the same data is fed to multiple input channels of the neural engine 314 such that multiple or all neural engines 314 receive the same data, or may operate in a unicast mode, in which different neural engines 314 receive different data), and (iv) transmit configuration commands to the planar engine 340 to instruct the planar engine 340 to program itself for operation in one of a number of operating modes.

[0065] The data stored in the buffer 334 of the neural processor circuit 218 may be a portion of image data, histograms of orientation gradient (HOG) data, audio data, metadata, output data 328 of the previous loop of the neural engine 314, and other processed data received from other components of the SOC component 204.

[0066] The data processor DMA 320 includes read circuitry that receives segments of input data from a source (e.g., system memory 230) for storage in a buffer 334, and write circuitry that forwards the data from the buffer 334 to a target component (e.g., system memory). In one embodiment, the direct memory access nature of the data processor DMA 320 allows the data processor DMA 320 to directly acquire and write data from a source (e.g., system memory 230) without the involvement of the CPU 208. The buffer 334 may be a direct memory access buffer that stores data of the machine learning model of device 100 without the involvement of the CPU 208.

[0067] Example Neural Engine Architecture

[0068] Figure 4A This is a block diagram of a neural engine 314 based on one implementation scheme. Specifically, Figure 4AThe diagram illustrates operations performed by the neural engine 314, including machine learning-friendly operations such as convolution, tensor product, and other operations that may involve significant computation in the first mode. To this end, the neural engine 314 receives input data 322, performs a multiplication-accumulation operation (e.g., convolution) on the input data 322 based on stored kernel data, performs further post-processing operations on the result of the multiplication-accumulation operation, and generates output data 328. The input data 322 and / or output data 328 of the neural engine 314 can be single-channel or span multiple channels.

[0069] The neural engine 314 may include an input buffer circuit 402, a computational kernel 416, a neural engine (NE) control 418, a kernel extraction circuit 432, an accumulator 414, and an output circuit 424, as well as other components. The neural engine 314 may include... Figure 4A The fewer components shown or including Figure 4A Other components not shown.

[0070] In a first mode, as a subset of data is received from the source, the input buffer circuit 402 stores that subset of data. The source may be the data processor circuit 318, the planar engine 340, or another suitable component. The input buffer circuit 402 sends an appropriate segment 408 of data for the current task or processing loop to the computational core 416 for processing. The input buffer circuit 402 may include a shifter 410 that shifts the read position of the input buffer circuit 402 to change the segment 408 of data sent to the computational core 416. By changing the segment of the input data 322 provided to the computational core 416 via shifting, the neural engine 314 can perform multiplication and accumulation on different segments of the input data 322 based on a smaller number of read operations. In one or more embodiments, the data of the neural processor circuit 218 includes data from different convolutional groups and / or input channels.

[0071] In the first mode, kernel extraction circuitry 432 receives kernel data 326 from kernel DMA 324 and extracts kernel coefficients 422. In one embodiment, kernel extraction circuitry 432 references a lookup table (LUT) and uses a mask to reconstruct the kernel from compressed kernel data 326 based on the LUT. The mask indicates the locations of unfilled zeros and the remaining locations of unfilled numbers in the reconstructed kernel. The kernel coefficients 422 of the reconstructed kernel are sent to compute core 416 to fill registers in the arithmetic circuitry of compute core 416. In other embodiments, kernel extraction circuitry 432 receives uncompressed kernel data 326 and determines kernel coefficients 422 without referencing a LUT or using a mask.

[0072] In the first mode, computation core 416 performs computational operations. For this purpose, computation core 416 may include arithmetic circuits OC0 to OCN and a post-processor 428. Each arithmetic circuit in OC0 to OCN stores the input value in a segment 408 of the input data and the corresponding kernel coefficient in kernel coefficient 422. In each arithmetic circuit in OC0 to OCN, the input value and the corresponding kernel coefficient are multiplied to generate a processed value 412.

[0073] In the first mode, accumulator 414 receives and stores the processed value 412 as a first result from the arithmetic circuitry. The processed value stored in accumulator 414 can be sent back as feedback information 419 for further multiplication and addition operations at the arithmetic circuitry, or sent to post-processor 428 for post-processing. In the first mode, accumulator 414, combined with the arithmetic circuitry, forms multiply-accumulate (MAC) 404. In one or more embodiments, accumulator 414 may have subunits, each of which sends data to a different component of neural engine 314. For example, during a processing loop, data stored in a first subunit of accumulator 414 is sent to MAC 404, while data stored in a second subunit of accumulator 414 is sent to post-processor 428.

[0074] In the first mode, postprocessor 428 performs further processing on the value 412 received from accumulator 414. Postprocessor 428 may perform operations including, but not limited to, applying linear functions (e.g., rectified linear unit (ReLU)), normalizing cross-correlation (NCC), combining the results of performing neural operations on 8-bit data into 16-bit data, and local response normalization (LRN). The result of such operations is output from postprocessor 428 to output circuitry 424 as a processed value 417. In some embodiments, processing at postprocessor 428 is bypassed. For example, data in accumulator 414 may be sent directly to output circuitry 424 for access by other components of neural processor circuitry 218.

[0075] The NE control 418 controls the operation of other components of the neural engine 314 based on the operating modes and parameters of the neural processor circuit 218. Depending on different operation modes (e.g., first mode or second mode), different convolution operation modes (e.g., group convolution mode or non-group convolution mode), or parameters (e.g., the number of input channels and the number of output channels), the neural engine 314 can operate on different input data in different sequences, return different values ​​from the accumulator 414 to the operation circuit, and perform different types of post-processing operations at the post-processor 428. To configure the components of the neural engine 314 to operate in a desired manner, the NE control 418 sends task commands, which can be included in the information 419, to the components of the neural engine 314. The NE control 418 may include a raster 430 that tracks the current task or processing loop being processed at the neural engine 314.

[0076] In the first mode, the rasterizer 430 can perform operations associated with dividing the input data into smaller units (segments) and regulating the processing of these smaller units via arithmetic circuitry and accumulator 414. The rasterizer 430 keeps track of the size and rank (e.g., group, working unit, input channel, output channel) of the input / output data segments and instructs components of the neural processor circuitry 218 to correctly process the segments of input data. For example, the rasterizer 430 operates shifter 410 in input buffer circuitry 402 to forward the correct segment 408 of input data 322 to MAC 404 and send the completed output data 328 to data buffer 334. Other components of the neural processor circuitry 218 (e.g., kernel DMA 324, buffer DMA 320, data buffer 334, planar engine 340) may also have their corresponding rasterizers for monitoring the division of input data and the parallel computation of individual segments of input data 322 in different components.

[0077] In the first mode, the output circuit 424 receives the processed value 417 from the post-processor 428 and interacts with the data processor circuit 318 to store the processed value 417 in the data processor circuit 318. For this purpose, the output circuit 424 may issue output data 328 in an order or format different from the order or format in which the processed value 417 is processed in the post-processor 428.

[0078] Components in the Neural Engine 314 can be configured by the NE control 418 and the Neural Task Manager 310 during the configuration cycle. To this end, the Neural Task Manager 310 sends configuration information to the Neural Engine 314 during the configuration cycle. Configurable parameters and modes may include, but are not limited to, the mapping between input data elements and kernel elements, the number of input channels, the number of output channels, the execution of output strides, and enabling / selecting post-processing operations at the post-processor 428.

[0079] Figure 4B This is a block diagram illustrating the operation of a neural engine 314 in a second mode according to one embodiment. In the second mode, the neural engine 314 performs parallel sorting operations (e.g., bitonic sort, top-k bitonic sort, counting sort, radix sort, batch merge sort, pairwise sort, etc.). To this end, the neural engine 314 receives input data 322, performs a portion of the parallel sorting operation (e.g., comparison operation) on the input data 322, performs further post-processing operations on the result of the parallel sorting operation, and generates output data 328.

[0080] In the second mode, as a subset of data is received from a source (e.g., data processor circuitry 318, planar engine 340, or another suitable component) into neural processor circuitry 218, input buffer circuitry 402 stores that subset of data. Input buffer circuitry 402 sends an appropriate segment 408 of data for the current task or processing loop to computational core 416 for processing. For example, data segment 408 may include a single vector or multiple vectors. Input buffer circuitry 402 may include a shifter 410 that shifts the read position of input buffer circuitry 402 to change the segment 408 of data sent to computational core 416. For example, at a first time, input buffer circuitry 402 may send a first vector to computational core 416, and at a second time, input buffer circuitry 402 may send a second vector to computational core 416. By shifting and changing segments of the input data 322 provided to the computation core 416, the neural engine 314 can perform various parts of parallel sorting operations on different segments of the input data 322 based on a small number of read operations to the data processor circuitry 318 or the system memory 230.

[0081] In the second mode, the kernel extraction circuit 432 is disabled by the NE control 418, while the arithmetic circuits OC0 to OCN perform sorting operations on the segments 408 of the input data. Specifically, each of the arithmetic circuits OC0 to OCN compares the input value with a previous input value to produce a processed value 412. In some embodiments, the processed value 412 may be a maximum or minimum value. In some embodiments, the processed value 412 may include a maximum value and metadata corresponding to the maximum value (e.g., an index indicating a larger value), or a minimum value and metadata corresponding to the minimum value (e.g., an index indicating a smaller value).

[0082] In the second mode, accumulator 414 receives and stores the result of the comparison operation as a second result from the arithmetic circuitry. The processed value 412 stored in accumulator 414 can be sent back as feedback information 419 for further comparison operations at arithmetic circuitry OC0 to OCN (e.g., for performing a subsequent part of a parallel sorting operation), or sent to post-processor 428 for post-processing. In the second mode, accumulator 414, combined with arithmetic circuitry OC0 to OCN, forms comparator-accumulator (CMP-AC) 434. In one or more embodiments, accumulator 414 may have sub-units, each of which sends data to a different part of neural engine 314. For example, during a processing loop, data stored in a first sub-unit of accumulator 414 is sent to CMP-AC 434, while data stored in a second sub-unit of accumulator 414 is sent to post-processor 428.

[0083] In the second mode, the post-processor 428, NE control 418, and output circuit 424 can perform operations related to... Figure 4A The operation is essentially the same in the first mode described herein. In the second mode, the rasterizer 430 also performs operations similar to those performed during the first mode, except that the rasterizer manages and tracks the division of the input data into smaller units (fragments) for sorting operations via CMP-AC 434 and accumulator 414.

[0084] Exemplary operational circuit and accumulator circuit diagram

[0085] Figure 5 This is a circuit diagram of the arithmetic circuit OC0 and accumulator circuit 414 of a neural engine 314 according to one embodiment. The arithmetic circuit OC0 and accumulator circuit 414 can be programmed to execute in a first mode or a second mode. In the first mode, the arithmetic circuit OC0 can perform a convolution multiply-accumulate operation on a segment of data 408, and the accumulator circuit 414 can operate to receive and store a first result 524 of the multiply-accumulate operation (e.g., a processed value 412). Although Figure 5 Only a single operational circuit OC0 is described, but other operational circuits OC1 to OCN may have the same structure as operational circuit OC0 and perform operations in the same way as operational circuit OC0.

[0086] In the second mode, the arithmetic circuit OC0 can perform at least a portion of a parallel sorting operation to sort data segments 408 according to their size. For these purposes, the arithmetic circuit OC0 may include a multiplexer 530, a multiplier 540, an adder 542, and a comparator circuit 550, as well as other components. The arithmetic circuit OC0 may include fewer components or may include... Figure 5Other components not shown. For example, in the second mode, the arithmetic circuit OC0 may not include comparator circuit 550, and adder 542 may be used as subtractor. Each component in the arithmetic circuit OC0 may be embodied as a circuit or a circuit combined with firmware or software.

[0087] During operation, the arithmetic circuit OC0 receives data segment 408 as input to the multiplexer 530. The select line 510 controls the output of the multiplexer 530. In a first mode, a control signal sent via the select line 510 instructs the multiplexer 530 to output data segment 408 to the multiplier 540. In a second mode, the control signal sent via the select line 510 instructs the multiplexer 530 to output data segment 408 to the comparator circuit 550. In some embodiments, the control signal is set by the neural task manager 310 and stored in the registers of the neural engine 314.

[0088] In the first mode, multiplier 540 multiplies data segment 408 with the corresponding kernel coefficient in kernel coefficient 422, producing a product value 520 corresponding to the multiplication of kernel coefficients and data. Multiplier 540 provides product value 520 to adder 542. Accumulator 414 provides the stored first result 522 (generated as the result of a previous addition operation at adder 542) to adder 542, causing adder 542 to add product value 520 to the stored first result 522. After adder 542 generates accumulated value 524, accumulated value 524 is sent back to accumulator 414 as first result 522. Then, in the next round, the updated first result 522 can be accumulated with another product value 520. This process is repeated until all data segments 408 and corresponding kernel coefficients 422 have been processed. When accumulator 414 does not have the stored first result 522 to provide to adder 542 (e.g., in the first round of processing), adder 542 passes product value 520 to accumulator 414 so that the first product value 520 can be stored as the initial first result 522.

[0089] In the second mode, comparator circuit 550 receives data segments 408 from multiplexer 530. Upon receiving a first data value from data segment 408, this first data value passes through comparator circuit 550 and is stored in accumulator 414. An initial index for sorting operations can be used to label the first data value. In the next round, a second data value is received as data segment 408 and fed to comparator circuit 550. Furthermore, a first data value 526 is retrieved from accumulator 414 and fed to comparator circuit 550. Comparator circuit 550 compares the first data value 526 with the second data value, updates their labeled indices to indicate which of the first or second data values ​​is larger, and forwards the second data value with its labeled index as a second result 528 to accumulator 414 for storage.

[0090] Subsequently, one of the stored vector / scalar values ​​526 is fed to the comparator circuit 550 and compared with the new data value received as the next data segment 408. This process of feeding one of the stored data values ​​526 to the comparator circuit 550 and comparing it with the new data value is repeated until the comparison operation is complete. The sequence of data values ​​to be compared can be set by the rasterizer 430, and a parallel sorting network can be implemented, as described below. Figure 7 Detailed description. With each round of comparisons, the indexes marked to each vector / scalar value are updated to reflect the order to be sorted.

[0091] In an alternative implementation, the data values ​​are not indexed. For example, comparator circuit 550 compares a first data value 526 and a second data value, and forwards the first data value 526 and the second data value in ascending or descending order as a second result 528 to accumulator 414 for storage.

[0092] In an implementation of the arithmetic circuit OC0 without comparator circuit 550, where adder 542 is used as subtractor, a first data value 526 is retrieved from accumulator 414 and fed to adder 542. Adder 542 subtracts the first data value 526 from a second data value. The sign bit of the result determines which of the first or second data values ​​should be forwarded to accumulator 414 as the second result 528 for storage. Subsequently, one of the stored vector / scalar values ​​526 is fed to adder 542 and subtracted from the new data value received as the next segment of data 408.

[0093] When the operational circuit OC0 performs its comparison operation in the second mode, one or more of the other operational circuits OC1 to OCN can perform their own comparisons in parallel.

[0094] Exemplary process for operating neural engine circuitry

[0095] Figure 6 This is a flowchart illustrating a method of operating the neural engine circuitry 314 in a first mode and a second mode according to one embodiment. The neural processor circuitry 218 (e.g., via the neural task manager 310) sets the mode of the neural engine circuitry 314.

[0096] When set to the first mode, the arithmetic circuitry of the neural engine circuit 314 (e.g., OC0 to OCN of MAC 404) receives 620 units of input data (e.g., data segments 408 received by each arithmetic circuit). The arithmetic circuits perform a 630-unit convolution multiplication-accumulation operation on the input data using kernels (e.g., kernel coefficients in kernel coefficient 422). Accumulator 414 receives and stores 640 accumulated values ​​as the first result of the multiplication-accumulation operation.

[0097] When set to the second mode, multiple computational circuits of the neural engine circuit 314 (e.g., OC0 to OCN of the CMP-AC 434) receive 620 input data (e.g., each computational circuit receives a segment 408 of data). The multiple computational circuits perform a portion of 635 parallel sorting operations on the input data. Accumulator 414 receives and stores a second result of 645 parallel sorting operations. The second result includes compared data values ​​and labeled indices indicating the sorted order of the data values.

[0098] The above references Figure 6 The implementation of the process described is merely illustrative. Furthermore, the order of the process may be modified or omitted.

[0099] Exemplary parallel sorting networks

[0100] Figure 7 This is a conceptual diagram illustrating an exemplary parallel sorting network 700 executed at neural processor circuitry 218 according to one embodiment. Figure 7 The parallel sorting network 700 shown is a bitonic sorting network that receives 16 data values ​​702 and simultaneously generates the maximum value 770 and the minimum value 780 of the input data at the end of the sorting operation.

[0101] exist Figure 7 In the example, the parallel sorting network 700 is embodied using one or more neural engine circuits 314. Multiple computational circuits (e.g., OC0 to OCN) within the one or more neural engine circuits 314 perform operations... Figure 7The arrows in the diagram indicate sorting (comparison) operations. Tasks 710, 720, 725, 730, 733, 735, 740, 741, 743, and 745 are executed sequentially to complete the parallel sorting operations. Each of these tasks can be executed by one or more neural engine circuits 314.

[0102] The sorting operation of Task 710 can be performed in a single loop of a neural engine circuit 314 by having eight operational circuits in its operational circuitry execute the sorting operation in parallel. Alternatively, the same operation can be performed in multiple loops using a smaller number of operational circuits in a neural engine circuit 314. If the number of sorting operations is large, the operational circuits in two or more neural engine circuits 314 can operate in parallel.

[0103] After task 710 is completed, one or more neural engine circuits 314 are updated to execute task 720 and perform the sorting operation defined in task 720. After tasks 720, 725, 730, 733, 735, 740, 741, 743, and 745, the indices marked to the data values ​​indicate the sorted order of the data values. The data value with the index indicating the highest number is output as the maximum number 770, and another data value with the index indicating the lowest number is output as the minimum number 780.

[0104] In alternative implementation schemes ( Figure 7 (Not shown in the diagram), the input data may include multiple elements (e.g., 16, 100, 256 elements, etc.). Each element may be a single data value (e.g., a single element in a vector), a vector, or multiple vectors that have undergone sorting operations performed by the neural engine 314.

[0105] While specific implementations and applications have been described and illustrated, it should be understood that the invention is not limited to the precise constructions and components disclosed herein, and that various modifications, alterations, and variations that will be apparent to those skilled in the art may be made to the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. A neural engine circuit, comprising: Neural engine control circuitry; Kernel extraction circuit, the kernel extraction circuit being configured as follows: In the first mode: Receive kernel data from a memory coupled to the neural engine circuitry; and Obtain the kernel from the kernel data; as well as In the second mode: Receive a command from the neural engine control circuit that disables the kernel extraction circuit; Multiple arithmetic circuits, wherein the multiple arithmetic circuits are configured as follows: Receive input data, In the first mode, the kernel obtained by the kernel extraction circuit performs a convolution multiplication-accumulation operation on the input data, and In the second mode, a portion of the parallel sorting operation is performed on the input data; and An accumulator circuit, coupled to the plurality of arithmetic circuits, is configured as follows: In the first mode, the first result of the multiplication-addition operation is received and stored, and In the second mode, the second result of the parallel sorting operation is received and stored.

2. The neural engine circuit of claim 1, wherein in the second mode, the second result of the parallel sorting operation is sent to the plurality of computation circuits to execute a subsequent portion of the parallel sorting operation.

3. The neural engine circuit according to claim 1, wherein the parallel sorting operation implements a bitone sorting network, and the bitone sorting network simultaneously generates the maximum and minimum values ​​of the input data.

4. The neural engine circuit of claim 1, wherein the accumulator circuit is further configured to receive and store, in the second mode, an index for the maximum value of the input data and an index for the minimum value of the input data.

5. The neural engine circuit of claim 1, wherein each of the computational circuits comprises: A multiplier circuit configured to receive a portion of the input data and a portion of the kernel in a first mode and perform a multiplication operation on the portion of the input data and the portion of the kernel; An adder circuit coupled to the multiplier circuit and the accumulator circuit, the adder circuit being configured to receive a stored first result and a value derived from the multiplication operation in the first mode and perform an addition operation on the first result and the value derived from the multiplication operation; and A comparator circuit coupled to the accumulator circuit is configured to receive a portion of the input data and a stored second result in the second mode and perform a comparison operation on the portion of the input data and the stored second result.

6. The neural engine circuit of claim 1 further includes an input buffer circuit coupled to the computational circuit to provide different sets of input data to the plurality of computational circuits in different cycles of the neural engine circuit.

7. The neural engine circuit of claim 1, wherein another neural engine circuit performs another portion of the parallel sorting operation in parallel with the neural engine circuit.

8. The neural engine circuit of claim 1, wherein the input data comprises a plurality of input data elements, and the plurality of arithmetic circuits are further configured to perform a portion of the parallel sorting operation on the plurality of input data elements in the second mode.

9. A method of operating a neural engine circuit, the method comprising: In the first mode: Kernel data is received from a memory coupled to the neural engine circuitry by the kernel extraction circuitry; as well as The kernel is obtained from the kernel data by the kernel extraction circuit. In the second mode: Receive a command from the neural engine control circuitry that disables the kernel extraction circuitry in the second mode; Input data is received by multiple computational circuits of the neural engine circuit; In the first mode, the plurality of arithmetic circuits perform convolution multiplication-addition operations on the input data using the kernel obtained by the kernel extraction circuit; In the second mode, a portion of the parallel sorting operation is performed on the input data by the plurality of arithmetic circuits; In the first mode, the accumulator circuit, which is coupled to the plurality of arithmetic circuits by the neural engine circuit, receives the first result of the multiplication-accumulation operation for storage; as well as In the second mode, the accumulator circuit receives the second result of the parallel sorting operation for storage.

10. The method of claim 9, further comprising: In the second mode, the second result of the parallel sorting operation is sent to the plurality of computing circuits to execute the subsequent part of the parallel sorting operation.

11. The method according to claim 9, wherein the parallel sorting operation implements a bitonic sorting network, and the bitonic sorting network simultaneously generates the maximum and minimum values ​​of the input data.

12. The method according to claim 9, further comprising: In the second mode, the accumulator circuit receives and stores the index for the maximum value of the input data and the index for the minimum value of the input data.

13. The method of claim 9, further comprising: In the first mode, a portion of the input data and a portion of the kernel are received by the multiplier circuit in each of the arithmetic circuits to perform a multiplication operation on the portion of the input data and the portion of the kernel; In the first mode, an adder circuit coupled to the multiplier circuit and the accumulator circuit receives a stored first result and a value derived from the multiplication operation, and performs an addition operation on the stored first result and the value derived from the multiplication operation. as well as In the second mode, a comparator circuit in each of the arithmetic circuits coupled to the accumulator circuit receives the portion of the input data and the stored second result to perform a comparison operation on the portion of the input data and the stored second result.

14. The method of claim 9, further comprising: The input buffer circuit, coupled to the plurality of computational circuits, provides different sets of input data to the computational circuits in different cycles of the neural engine circuit.

15. The method of claim 9, wherein another neural engine circuit performs another portion of the parallel sorting operation in parallel with the neural engine circuit.

16. The method of claim 9, wherein the input data comprises a plurality of input data elements, and the plurality of arithmetic circuits are further configured to perform a portion of the parallel sorting operation on the plurality of input data elements in the second mode.

17. An electronic device comprising: At least one neural engine circuit, the at least one neural engine circuit comprising: Neural engine control circuitry; Kernel extraction circuit, the kernel extraction circuit being configured as follows: In the first mode: Receive kernel data from a memory coupled to the at least one neural engine circuit; and Obtain the kernel from the kernel data; and In the second mode: Receive a command from the neural engine control circuit that disables the kernel extraction circuit; Multiple arithmetic circuits, wherein the multiple arithmetic circuits are configured as follows: Receive input data, In the first mode, the kernel obtained by the kernel extraction circuit performs a convolution multiplication-accumulation operation on the input data, and In the second mode, a portion of the parallel sorting operation is performed on the input data; and An accumulator circuit, coupled to the plurality of arithmetic circuits, is configured as follows: In the first mode, the first result of the multiplication-addition operation is received and stored, and In the second mode, the second result of the parallel sorting operation is received and stored.

18. The electronic device of claim 17, wherein in the second mode, the second result of the parallel sorting operation is sent to the plurality of arithmetic circuits to perform a subsequent portion of the parallel sorting operation.

19. The electronic device of claim 17, wherein each of the arithmetic circuits comprises: A multiplier circuit configured to receive a portion of the input data and a portion of the kernel in a first mode and perform a multiplication operation on the portion of the input data and the portion of the kernel; An adder circuit coupled to the multiplier circuit and the accumulator circuit, the adder circuit being configured to receive a stored first result and a value derived from the multiplication operation in the first mode and perform an addition operation on the first result and the value derived from the multiplication operation; and A comparator circuit coupled to the accumulator circuit is configured to receive a portion of the input data and a stored second result in the second mode and perform a comparison operation on the portion of the input data and the stored second result.

20. The electronic device of claim 17, wherein the neural engine circuit further comprises an input buffer circuit coupled to the plurality of computational circuits to provide different sets of input data to the computational circuits in different cycles of the at least one neural engine circuit.

Images