Splitting input data for processing in a neural network processor

Abstract

Embodiments of the present disclosure relate to splitting input data into smaller units for loading into data buffers and neural engines in a neural processor circuit for performing neural network operations. Large input data is split into slices, and each slice is further split into tiles. The tiles are uploaded from an external source to a data buffer located inside the neural processor circuit but outside the neural engines. Each tile is further split into work units sized for storage in input buffer circuits within each neural engine. Input data stored in the data buffer and the input buffer circuits is reused by the neural engines to reduce re-fetching of input data. The operation of splitting the input data is performed at various components of the neural processor circuit under the management of a rasterizer provided in the components.

Description

Background Technology 1. Technical Field

[0002] This disclosure relates to circuitry for performing neural network operations, and more specifically to rasterization of input data for processing at a neural network processor.

[0003] 2. Relevant Technical Descriptions

[0004] Artificial neural networks (ANNs) are computational systems or models that use a set of connected nodes to process input data. ANNs are typically organized into layers, with different layers performing different types of transformations on their inputs. Extensions or variants of ANNs such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep belief networks (DBNs) have received considerable attention. These computational systems or models typically involve a wide range of computational operations, including multiplication and accumulation. For example, CNNs are a class of machine learning techniques that primarily use convolutions between input data and kernel data; convolutions can be decomposed into multiplication and accumulation operations.

[0005] These machine learning systems or models can be configured differently depending on the type of input data and the operations to be performed. Such varied configurations would include, for example, preprocessing operations, the number of channels in the input data, the kernel data to be used, the nonlinear functions to be applied to the convolution results, and the application of various post-processing operations. Instantiating and executing machine learning systems or models with various configurations using a central processing unit (CPU) and its main memory is relatively straightforward, as such systems or models can be instantiated simply by updating the code. However, relying solely on the CPU for the various operations of these machine learning systems or models would consume significant CPU bandwidth and increase overall power consumption. Summary of the Invention

[0006] The implementation relates to a neural processor circuit having raster circuitry located at various components to coordinate and track operations. The neural processor circuit may include a data reader, a data buffer, and a neural engine. The data reader includes raster circuitry that instructs the data reader to read segments of input data from a source external to the neural processor circuit. The data buffer stores the received segments of the input data. The data buffer includes another raster circuitry for generating and transmitting work units. Each work unit corresponds to a portion of the input data segment stored in the data buffer. The neural engine receives the work unit and generates an output value based on a multiply-accumulate operation performed on a portion of the work unit using the corresponding kernel.

[0007] In one or more embodiments, each neural engine in the neural engine includes an additional grating circuit that instructs each neural engine in the neural engine to shift the portion of the working unit to be multiplied with the core at different cycles. Attached Figure Description

[0008] Figure 1 It is a high-level diagram of an electronic device according to the implementation plan.

[0009] Figure 2 This is a block diagram illustrating components in an electronic device according to one embodiment.

[0010] Figure 3 This is a block diagram illustrating a neural processor circuit according to one embodiment.

[0011] Figure 4 This is a block diagram of a neural engine in a neural processor circuit according to one implementation scheme.

[0012] Figure 5 This is a conceptual diagram illustrating a loop for processing input data at a neural processor circuit according to one embodiment.

[0013] Figure 6 This is a conceptual diagram illustrating how input data is segmented into slices, tiles, and work cells according to one implementation scheme.

[0014] Figure 7 This is a diagram illustrating the programming of a grating in a component of a neural processor circuit according to one embodiment.

[0015] Figure 8 This is a flowchart illustrating a method for processing input data in a neural processor circuit according to one embodiment.

[0016] For illustrative purposes only, the accompanying drawings and detailed descriptions depict various non-limiting embodiments. Detailed Implementation

[0017] Reference will now be made in detail to the embodiments, examples of which are shown in the accompanying drawings. Numerous specific details are shown in the following detailed description to provide a full understanding of the various described embodiments. However, the embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the various aspects of the embodiments.

[0018] This disclosure relates to splitting input data into smaller data units for loading into a data buffer and a neural engine within a neural processor circuitry used to perform neural network operations. Large input data is split into slices, and each slice is further split into tiles. These tiles are uploaded from an external source to a data buffer located inside the neural processor circuitry but outside the neural engine. Each slice is then further split into working units, the size of which is configured for storage in an input buffer circuitry within each neural engine. The input data stored in the data buffer and the input buffer circuitry is reused by the neural engine to reduce the re-fetching of input data. This input data splitting operation is performed under the management of rasterizers located within various components of the neural processor circuitry.

[0019] Exemplary electronic devices

[0020] This document describes implementations of electronic devices, user interfaces for such devices, and related processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions such as a personal digital assistant (PDA) and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Devices, Apple Equipment and Device. Alternatively, other portable electronic devices, such as wearable devices, laptops, or tablets, may be used. In some embodiments, the device is not a portable communication device, but a desktop computer or other computing device not designed for portable use. In some embodiments, the disclosed electronic device may include a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). The following is combined with… Figure 1 The described example electronic device (e.g., device 100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0021] Figure 1This is a high-level diagram of an electronic device 100 according to one embodiment. Device 100 may include one or more physical buttons, such as a "home" button or a menu button 104. Menu button 104 is used, for example, to navigate to any application in a set of applications running on device 100. In some embodiments, menu button 104 includes a fingerprint sensor for recognizing a fingerprint on menu button 104. The fingerprint sensor can be used to determine whether the finger on menu button 104 has a fingerprint that matches a fingerprint stored for unlocking device 100. Alternatively, in some embodiments, menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touchscreen.

[0022] In some embodiments, device 100 includes a touchscreen 150, a menu button 104, a push-button 106 for powering the device on / off and for locking the device, a volume control button 108, a subscriber identity module (SIM) card slot 110, a headset jack 112, and a docking / charging external port 124. The push-button 106 can be used to power the device on / off by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 includes various components, including but not limited to memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include Figure 1 Components not shown in the diagram.

[0023] Device 100 is merely one example of an electronic device, and device 100 may have more or fewer components than those listed above, some of which may be combined into a single component or have different configurations or arrangements. The various components of device 100 listed above are embodied in hardware, software, firmware, or combinations thereof, including one or more signal processing and / or application-specific integrated circuits (ASICs).

[0024] Figure 2This is a block diagram illustrating components in a device 100 according to one embodiment. Device 100 can perform various operations, including image processing. For this and other purposes, device 100 may include an image sensor 202, a system-on-a-chip (SOC) component 204, a system memory 230, permanent memory (e.g., flash memory) 228, an orientation sensor 234, and a display 216, as well as other components. Figure 2 The components shown are merely illustrative. For example, device 100 may include... Figure 2 Other components not shown (such as speakers or microphones). Additionally, some components (such as orientation sensor 234) may be omitted from device 100.

[0025] Image sensor 202 is a component for capturing image data and can be implemented as, for example, a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, camera, camcorder, or other device. Image sensor 202 generates raw image data, which is sent to SOC component 204 for further processing. In some embodiments, the image data processed by SOC component 204 is displayed on display 216, stored in system memory 230, permanent memory 228, or transmitted to a remote computing device via a network connection. The raw image data generated by image sensor 202 may be a Bayer color kernel array (CFA) pattern (hereinafter also referred to as a "Bayer pattern").

[0026] Motion sensor 234 is a component or set of components used to sense the motion of device 100. Motion sensor 234 can generate sensor signals indicating the orientation and / or acceleration of device 100. The sensor signals are sent to SOC component 204 for various operations, such as turning on device 100 or rotating an image displayed on display 216.

[0027] Display 216 is a component for displaying images generated by SOC component 204. Display 216 may include, for example, a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) device. Based on data received from SOC component 204, display 216 may display various images, such as menus, selected operating parameters, images captured by image sensor 202 and processed by SOC component 204, and / or other information (not shown) received from the user interface of device 100.

[0028] System memory 230 is a component used to store instructions executed by SOC component 204 and to store data processed by SOC component 204. System memory 230 can be embodied in any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or combinations thereof. In some embodiments, system memory 230 can store pixel data or other image data or statistics in various formats.

[0029] Permanent memory 228 is a component used to store data in a non-volatile manner. Permanent memory 228 retains data even when power is unavailable. Permanent memory 228 may be manifested as read-only memory (ROM), flash memory, or other non-volatile random access memory devices.

[0030] SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing procedures. SOC component 204 may include, among other sub-components, an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a sensor interface 212, a display controller 214, a neural processor circuit 218, a graphics processor (GPU) 220, a memory controller 222, a video encoder 224, a memory controller 226, and a bus 232 connecting these sub-components. SOC component 204 may include... Figure 2 The sub-components shown have more or fewer sub-components.

[0031] The ISP 206 is the hardware that performs each stage of the image processing pipeline. In some implementations, the ISP 206 may receive raw image data from the image sensor 202 and process the raw image data into a form available to other sub-components of the SOC component 204 or components of the device 100. The ISP 206 may perform various image processing operations, such as image translation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as referenced below. Figure 3 Detailed description.

[0032] CPU 208 can be implemented using any suitable instruction set architecture and can be configured to execute instructions defined in that instruction set architecture. CPU 208 can be a general-purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, RISC, ARM, or MIPS ISA, or any other suitable ISA. Although Figure 2A single CPU is shown, but SOC component 204 may include multiple CPUs. In a multiprocessor system, each CPU may collectively implement the same ISA, but this is not required.

[0033] The graphics processing unit (GPU) 220 is a graphics processing circuit for executing graphics data. For example, the GPU 220 may render objects to be displayed in a frame buffer (e.g., a frame buffer that includes pixel data for the entire frame). The GPU 220 may include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of some graphics operations.

[0034] Neural processor circuit 218 is a circuit that performs various machine learning operations based on computations including multiplication, addition, and accumulation. Such computations can be arranged to perform, for example, convolutions of input data and kernel data. Neural processor circuit 218 is a configurable circuit that performs these operations in a fast and efficient manner, while alleviating the resource-intensive operations associated with CPU 208 and neural network operations. Neural processor circuit 218 can receive input data from sensor interface 302, image signal processor 206, system memory 230, or other sources such as network interface 210 or GPU 220. The output of neural processor circuit 218 can be provided to various components of device 100, such as image signal processor 206, system memory 230, or CPU 208 for various operations. The structure and operation of neural processor circuit 218 are referenced below. Figure 3 Detailed description.

[0035] Network interface 210 is a sub-component that supports the exchange of data between device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data can be received from other devices via network interface 210 and stored in system memory 230 for subsequent processing (e.g., via networks such as those described below). Figure 3 The image signal processor 206 (as discussed in the text) and display are connected via a network interface 210. The network may include, but is not limited to, a local area network (LAN) (e.g., Ethernet or a corporate network) and a wide area network (WAN). Image data received via the network interface 210 can be processed by the ISP 206.

[0036] Sensor interface 212 is a circuit used to communicate with motion sensor 234. Sensor interface 212 receives sensor information from motion sensor 234 and processes the sensor information to determine the orientation or movement of device 100.

[0037] Display controller 214 is a circuit used to send image data to be displayed on display 216. Display controller 214 receives image data from ISP 206, CPU 208, graphics processor or system memory 230, and processes the image data into a format suitable for display on display 216.

[0038] The memory controller 222 is circuitry for communicating with the system memory 230. The memory controller 222 can read data from the system memory 230 for processing by the ISP 206, CPU 208, GPU 220, or other sub-components of the SOC component 204. The memory controller 222 can also write data to the system memory 230 received from various sub-components of the SOC component 204.

[0039] The video encoder 224 is hardware, software, firmware, or a combination thereof used to encode video data into a format suitable for storage in permanent memory 128, or to pass data to network interface 210 for transmission over a network to another device.

[0040] In some implementations, one or more sub-components of SOC component 204, or some functions of these sub-components, may be performed by software components executing on ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, permanent memory 228, or in another device communicating with device 100 via network interface 210.

[0041] Image or video data can flow through various data paths within SOC component 204. In one example, raw image data can be generated from image sensor 202 and processed by ISP 206, then sent to system memory 230 via bus 232 and memory controller 222. After the image data is stored in system memory 230, it can be accessed by video encoder 224 for encoding, or by display 116 for display via bus 232.

[0042] Example Neural Processor Circuit

[0043] The neural processor circuit 218 is a configurable circuit that performs neural network operations on input data, at least based on kernel data 340. For this purpose, the neural processor circuit 218 may include, among other components, a neural task manager 310, multiple neural engines 314A to 314N (collectively referred to below as "multiple neural engines 314" and also individually as "neural engine 314"), kernel direct memory access (DMA) 324, a data buffer 318, and a buffer DMA 320. The neural processor circuit 218 may include... Figure 3 Other components not shown.

[0044] Each neural engine in neural engine 314 performs computational operations of neural network operations in parallel. Depending on the operation load, the entire set of neural engines 314 may be operated, or only a subset of neural engines 314 may be operated, while the remaining neural engines 314 are placed in a power-saving mode to conserve power. Each neural engine in neural engine 314 includes components for storing one or more kernels, for performing multiplication and accumulation operations, and for post-processing to generate output data 328, as described below. Figure 4 Detailed description. An example of a neural network operation is the convolution operation.

[0045] The neural task manager 310 manages the overall operation of the neural processor circuitry 218. The neural task manager 310 may receive a list of tasks from a compiler executed by the CPU 208, store tasks in its task queue, select tasks to be executed, and send instructions to other components of the neural processor circuitry 218 for executing the selected tasks. The neural task manager 310 may also perform task switching upon detecting events such as receiving instructions from the CPU 208. In one or more embodiments, the neural task manager 310 sends raster information to components of the neural processor circuitry 218 to enable each of these components to track, retrieve, or process appropriate portions of input data and kernel data, as referenced below. Figures 5 to 7 Detailed description. Although in Figure 3 The neural task manager 310 is shown as part of the neural processor circuitry 218, but the neural task manager 310 may be a component external to the neural processor circuitry 218.

[0046] Kernel DMA 324 is a read circuit that retrieves kernel data from a source (e.g., system memory 230) and sends kernel data 326A to 326N to each neural engine in neural engine 314. The kernel data represents information from which kernel elements can be extracted. In one embodiment, the kernel data may be a compressed format that is decompressed at each neural engine in neural engine 314. Although in some cases the kernel data provided to each neural engine in neural engine 314 may be the same, in most cases the kernel data provided to each neural engine in neural engine 314 is different.

[0047] Data buffer 318 is a temporary storage device for storing data associated with neural network operations. In one embodiment, data buffer 318 is embodied as a memory accessible by all neural engines 314. Data buffer 318 may store input data 322A to 322N for feeding to corresponding neural engines 314A to 314N, and outputs from each of neural engines 314A to 314N for feeding back to neural engine 314 or sending to target circuitry (e.g., system memory 230). The operation of data buffer 318 is coordinated with other components of neural processor circuitry 218 such that input data and intermediate data stored in data buffer 318 are reused across multiple operations at neural engines 314, thereby reducing data transfers to and from system memory 230. Data buffer 318 may operate in broadcast or unicast mode, in which data input data from all input channels is fed to all neural engines 314, and in unicast mode, data input data from a subset of input channels is fed to each neural engine 314.

[0048] The input data 322 stored in the data buffer 318 includes image data, a histogram of orientation gradient (HOG) data, audio data, metadata, the output data 328 of the previous cycle of the neural engine 314, and other processed data received from other components of the SOC component 204.

[0049] The buffer DMA 320 includes read circuitry for receiving a portion (e.g., a block) of input data from a source (e.g., system memory 230) for storage in data buffer 318, and write circuitry for forwarding data from data buffer 138 to a destination (e.g., system memory).

[0050] Example Neural Engine Architecture

[0051] Figure 4 This is a block diagram of a neural engine 314 according to one embodiment. The neural engine 314 performs various operations to facilitate neural network operations, such as convolution, spatial pooling, and local response normalization. The neural engine 314 receives input data 322, performs a multiplication-accumulation operation (e.g., convolution) on the input data 322 based on stored kernel data, performs further post-processing operations on the result of the multiplication-accumulation operation, and generates output data 328. The input data 322 and / or output data 328 of the neural engine 314 can be a single channel or multiple channels.

[0052] The neural engine 314 may include, among other components, an input buffer circuit 402, a computational core 416, a neural engine (NE) control 418, a kernel extraction circuit 432, an accumulator 414, and an output circuit 424. The neural engine 314 may include... Figure 4 Other components not shown.

[0053] Input buffer circuit 402 is a circuit that stores a portion of input data 322 upon receipt from data buffer 318 and sends an appropriate portion 408 of the input data 402 for processing by computation core 416. Input buffer circuit 402 includes a shifter 410 that shifts the read position of input buffer circuit 402 to change the portion 408 of the input data sent to computation core 416. By changing the portion of the input data provided to computation core 416 via shifting, neural engine 314 can perform multiplication and accumulation on different portions of the input data based on a smaller number of read operations. In one or more embodiments, input data 322 includes data from differential convolution groups and / or input channels.

[0054] Kernel extraction circuit 432 is a circuit that receives kernel data 326 from kernel DMA 324 and extracts kernel coefficients 422. In one embodiment, kernel extraction circuit 432 references a lookup table (LUT) and uses a mask to reconstruct the kernel from compressed kernel data 326. The mask indicates the locations of unfilled zeros and the remaining locations of unfilled numbers in the reconstructed kernel. The kernel coefficients 422 of the reconstructed kernel are sent to compute core 416 to populate registers in the multiply-accumulate (MAD) circuitry of compute core 416. In other embodiments, kernel extraction circuit 432 receives kernel data in uncompressed format and determines the kernel coefficients without referencing a LUT or using a mask.

[0055] The computational core 416 is a programmable circuit that performs computational operations. For this purpose, the computational core 416 may include MAD circuits MAD0 to MADN and a post-processor 428. Each of the MAD circuits MAD0 to MADN stores an input value in an input data portion 408 and a corresponding kernel coefficient in a kernel coefficient 422. In each of the MAD circuits, the input value and the corresponding kernel coefficient are multiplied to generate a processed value 412.

[0056] Accumulator 414 is a memory circuit that receives and stores processed values ​​412 from the MAD circuit. The processed values ​​stored in accumulator 414 can be sent back as feedback information 419 for further multiplication and addition operations at the MAD circuit, or sent to post-processor 428 for post-processing. Accumulator 414, combined with the MAD circuit, forms multiplication accumulator (MAC) 404. In one or more embodiments, accumulator 414 may have sub-units, each of which sends data to a different part of neural engine 314. For example, during a processing cycle, data stored in a first sub-unit of accumulator 414 is sent to the MAC circuit, while data stored in a second sub-unit of accumulator 414 is sent to post-processor 428.

[0057] Postprocessor 428 is circuitry that performs further processing on the value 412 received from accumulator 414. Postprocessor 428 may perform operations including, but not limited to, applying linear functions (e.g., rectified linear unit (ReLU)), normalizing cross-correlation (NCC), combining the results of performing neural operations on 8-bit data into 16-bit data, and local response normalization (LRN). The result of such operations is output from postprocessor 428 to output circuitry 424 as a processed value 417.

[0058] The NE control 418 controls the operation of other components of the neural engine 314 based on the operating modes and parameters of the neural processor circuit 218. Depending on different operating modes (e.g., group convolution mode or non-group convolution mode) or parameters (e.g., the number of input channels and the number of output channels), the neural engine 314 can operate on different input data in different sequences, return different values ​​from the accumulator 414 to the MAD circuit, and perform different types of post-processing operations at the post-processor 428. To configure the components of the neural engine 314 to operate in a desired manner, the NE control 418 sends control signals to the components of the neural engine. The NE control 418 may also include a raster 430 that tracks the current task or processing loop being processed at the neural engine 314, as referenced below. Figures 5 to 7 Detailed description.

[0059] The output circuit 424 receives the processed value 417 from the post-processor 428 and interacts with the data buffer 318 to store the processed value 417 in the data buffer 318. For this purpose, the output circuit 424 may issue output data 328 in an order or format different from the order or format in which the processed value 417 is processed in the post-processor 428.

[0060] Components in the Neural Engine 314 can be configured by the NE control 418 and the Neural Task Manager 310 during the configuration cycle. To this end, the Neural Task Manager 310 sends configuration information to the Neural Engine 314 during the configuration cycle. Configurable parameters and modes may include, but are not limited to, the mapping between input data elements and kernel elements, the number of input channels, the number of output channels, the execution of output strides, and enabling / selecting post-processing operations at the post-processor 428.

[0061] The data is segmented for processing at the neural processor circuitry.

[0062] Input data is typically divided into smaller data pieces for parallel processing at multiple neural engines 314. Multiple operation loops are usually executed to generate the output of the task associated with the neural network. A compiler, executed by CPU 208, analyzes the hierarchical structure and nodes of the neural network and determines how to segment the input data based on the hardware constraints of the neural processor circuitry 218. One of the compiler's functions is to determine how to divide the input data into smaller data units for processing at the neural engines 314 and how to iteratively process them in loops to produce results for the task.

[0063] Figure 5 This is a conceptual diagram illustrating a loop for processing input data at neural processor circuitry 218 according to one embodiment. If group convolution involving multiple groups of convolutions is used, the outermost loop represents the processing of the groups of convolutions. Group convolution is a convolution in which the input data of the input channels in each group is used only to generate the output data of the output channels of that group, but not to generate the output data of the output channels of other groups. Therefore, each group of group convolutions can be viewed as a separate convolution operation.

[0064] Each convolutional group contains a processing loop for slices of input data. The entire input data used for the convolution operation is segmented into multiple slices in an overlapping manner, such as... Figure 6 As shown. The overlapping portions 602, 604, and 606 represent the parts of the input data that are over-extracted in two adjacent slices to provide spatial support for the corresponding kernel. The second outermost loop performs a convolution operation on each slice of the input data. Inside the slice's loop is the processing loop for the slice's tiles. Each tile is segmented into multiple tiles, such as... Figure 6 As shown. The overlapping portions 608, 610, 612, and 614 are the parts of the input data in slice 4 that are over-captured in two adjacent tiles to provide space support for the corresponding kernel. The rightmost tile will typically have a width smaller than the other tiles in the slice. In one implementation, the input data for each tile is loaded onto data buffer 318 during a read cycle and is repeatedly used for operations in the tile's processing loop. The processing loop for the tile is a processing loop for work units. Each tile is segmented into multiple work units, such as... Figure 6 As shown. A working unit is a portion of the input data having dimensions that produce an output value fitted to the accumulator 414 of the neural engine 314 during a single cycle of computation kernel 416. Although the shape of each working unit is... Figure 6 The work unit is shown as a horizontal bar, but its shape can vary depending on the shape and size of the tile. The work unit also has an overlapping portion representing the acquired data to support the corresponding kernel. In particular, if the tile is tall, the work unit for the last tile used for slicing can have a vertical bar shape. In one or more embodiments, each work unit is 256 bytes in size. In such embodiments, for example, the work unit can be shaped as one of 16×16, 32×8, 64×4, 128×2, or 256×1 dimensions.

[0065] For each working unit, an internal processing loop can be provided for an Output Channel Group (OCG). The number of output channels generated by a single cycle of computation core 416 for a given working unit is called the OCG. Depending on the operating mode, each neural engine 314 can process output data for a different number of output channels (e.g., 8 channels, 32 channels) for input data to be loaded into its input buffer circuitry 402 in a single pass.

[0066] For each output channel group, an internal processing loop can be provided for the input channel (Cin). If an input step is implemented to skip certain input data, a loop for the sub-input channel (Sub-Cin) can be provided within the processing loop for the input channel (Cin).

[0067] For each input channel or each sub-input channel, an internal loop is provided to process the horizontal spatial support of the kernel and the vertical support within each horizontal spatial support. Spatial support refers to the input data used for convolution with the kernel and includes over-acquiring input data for performing convolution at the edges of the input data.

[0068] Overfetching refers to acquiring additional input data within the current slice, tile, or cell of work, so that input data of appropriate dimensions can be provided for convolution with the kernel. In one or more implementations, overfetching is performed vertically between slices to obtain additional rows of input data (in... Figure 6 As shown in the diagram, overlapping portions 602, 604, and 606), horizontal overlays are performed between the tiles to obtain additional columns for input data (in...). Figure 6 The overlapping portions (608, 606, 612, 614) are shown in the figure, and additional rows are obtained by vertically acquiring input data between the working cells within the block.

[0069] For each space support used by the kernel, an internal processing loop is provided for the output channel (OC) to generate output data for each output channel (Cout). In the case of spatial upsampling implemented by output strut, an additional internal loop is provided for processing each sub-output channel. Kernel coefficients and MAC operations are loaded within the loop of the output channel (OC) or sub-output channel (if output strut is implemented) to generate output data for the output channel (OC) or sub-output channel.

[0070] Figure 5 The nested loop structure shown is merely illustrative. Loops can be omitted, added, or structured differently depending on various factors. For example, if only a single convolutional group is used, the outermost loop can be removed. Furthermore, the loop structures used for horizontal and vertical spatial support can be reversed.

[0071] In one or more implementations, as referenced above Figure 5 and Figure 6 The operations associated with dividing the input space into smaller units and processing these smaller units are performed by rasterizers 714, 718, 720, and 722 in various components of the neural processor circuitry 218. Rasterizers are circuits in various components of the neural processor circuitry 218 that maintain tracking segments of input / output data (e.g., groups, work units, input channels, output channels) and instruct components of the neural processor circuitry to correctly process the segments of input data. For example, rasterizer 720 in buffer DMA 320 tracks tiles and slices received from system memory 230, while rasterizer 718 in data buffer 318 sequentially broadcasts work units for processing by neural engine 314. Rasterizer 724 in kernel DMA 324 determines which kernels to receive and assigns them to neural engine 314, while rasterizer 714 in neural engine 314 operates shifter 410 in input buffer circuitry 402 to forward the correct portion 408 of input data to MAC 404 and sends the completed output data 328 to data buffer 318.

[0072] Figure 7This diagram illustrates the programming of rasterizers 714, 718, 720, and 722 in components 314, 318, 320, and 322 of a neural processor circuit 218 according to one embodiment. To perform their functions, each of the rasterizers 714, 718, 720, and 722 receives task information 710 instructing how input data and / or kernel data are segmented and processed by each component of the neural processor circuit 218. This task information includes details about the current layer (e.g., the dimensions of the input and output data, the dimensions of the associated kernel, and the type of padding at input data boundaries). The rasterizers 714, 718, 720, and 722 may also receive constraints on their operation (e.g., whether tile widths are allowed or not allowed to exceed a threshold).

[0073] By providing rasterizers in different components of the neural processor circuit 218, the overhead of data transfer between components of the neural processor circuit 218 can be reduced. If a single central rasterizer were provided to control the different components of the neural processor circuit 218, kernel data, input data, and output data might need to be transferred between these components to identify associated locations in the task loop, such as convolutional groups, tiles, slices, working units, input channels, and output channels. By using distributed rasterizers, separate metadata is not required to transfer kernel data, input data, and output data between components of the neural processor circuit 218.

[0074] Example process at the neural engine architecture

[0075] Figure 8 This is a flowchart illustrating a method for processing input data in neural processor circuitry 218 according to one embodiment. After the neural task manager 310 programs the rasterizers 714, 718, 720, and 722, the operation of buffer DMA 320 is initiated by rasterizer 720 instructing buffer DMA 320 804 to receive a block of input data from system memory 230. The block received by buffer DMA 320 is stored 806 in data buffer 318.

[0076] Then, the rasterizer 718 in the data buffer 318 instructs the data buffer 318 to send a work unit to one or more neural engines 314. The work unit is then stored in the input buffer circuit 402 of one or more neural engines 314.

[0077] In one or more embodiments, input buffer circuitry 402 selects 816 a portion of the working unit to be sent to MAC 404 for a multiply-accumulate operation. MAC 404 then performs the multiply-accumulate operation 820 on the selected portion of the working unit using the corresponding kernel. It is then determined whether the entire working unit 824 has been processed at one or more neural engines 314. If not, the selected portion of the working unit is shifted 828 by shifter 410, and another round of multiply-accumulate operation 820 is performed.

[0078] If it is determined that the entire work unit 824 has been processed, then it proceeds to determine whether all work units in the block have been processed 832. If not, the process advances 836 to the next work unit by causing the data buffer 318 to send the next work unit 808 to one or more neural engines 314, and the subsequent process is repeated.

[0079] If it is determined that all working units in the block 832 are processed by the neural engine 314, the process continues to determine whether all blocks used for input data 840 have been processed. If not, the process advances to the next block 844 by having the raster 720 instruct the buffer DMA 320 804 to receive the next block from the system memory 230 and repeating the subsequent process.

[0080] If it is determined that all tiles of the 840 input data have been processed, the process ends for the current input data. The process can then be repeated to process the next input data or proceed to the next task.

[0081] The above references Figure 8 The implementation scheme of the process described is merely illustrative. Other loops may be implemented, as referenced above. Figure 5 The order of this process can be modified or omitted.

[0082] While specific implementations and applications have been described and illustrated, it should be understood that the invention is not limited to the precise constructions and components disclosed herein, and that various modifications, alterations, and variations that will be apparent to those skilled in the art may be made to the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. A neural processor circuit, comprising: Multiple neural engines, each including input buffer circuitry; A data buffer circuit is configured to provide a corresponding working unit of a plurality of working units to a corresponding input buffer circuit of at least one of the plurality of neural engines, wherein each corresponding working unit corresponds to a portion of a segment of input data; and A raster circuit, configured to instruct the corresponding input buffer circuit of the at least one neural engine to shift a portion of the corresponding working unit. The at least one of the neural engines includes a multiply-accumulate circuit configured to perform digital multiplication and addition operations on the shifted portion of the corresponding working unit, which is shifted by the corresponding input buffer circuit, using a kernel.

2. The neural processor circuit of claim 1 further includes a neural task manager configured to send task information to the raster circuit, the task information indicating at least how the input data is segmented into the plurality of working units.

3. The neural processor circuit of claim 1, wherein the segment of the input data is a subset of: a slice of the input data, a slice of the input data that vertically divides the input data, and a segment that horizontally divides each slice of the slice.

4. The neural processor circuit of claim 1, wherein at least two of the plurality of working units have different shapes.

5. The neural processor circuit of claim 1, wherein the segment of said input data has a first size; and The data buffer circuit is further configured to: A corresponding second working unit corresponding to a second segment of the input data is provided to the corresponding input buffer circuit, wherein the second segment of the input data has a second size.

6. The neural processor circuit of claim 1, wherein the multiply-accumulate circuit is further configured to: The kernel generates processed numerical values ​​using the multiplication and addition operations of the numbers based on the shifted portion of the corresponding working unit, wherein the at least one neural engine further includes an accumulator configured to store the processed numerical values.

7. The neural processor circuit of claim 6, wherein the multiply-accumulate circuit is further configured to: Perform a second number multiplication and addition operation on the processed numerical value.

8. A method of operating a neural processor circuit, comprising: The neural processor circuit provides multiple neural engines with working units corresponding to segments of input data from the data buffer circuit; The working unit is stored in the input buffer circuit of at least one of the plurality of neural engines; The input buffer circuit shifts a portion of the working unit; as well as At least one of the plurality of neural engines uses a kernel to perform digital multiplication and addition operations on the shifted portion of the working unit that has been shifted by the input buffer circuit.

9. The method of claim 8, further comprising sending task information from a neural task manager in the neural processor circuitry to a raster circuitry, the task information indicating at least how the input data is segmented into the plurality of working units.

10. The method of claim 8, wherein the segment of the input data is a subset of: a slice of the input data, a slice of the input data that vertically divides the input data, and a segment that horizontally divides each slice of the slice.

11. The method of claim 8, wherein at least two of the plurality of working units have different shapes.

12. The method of claim 8, wherein the segment of the input data has a first size; and A second working unit corresponding to a second segment of the input data from the data buffer circuit is provided to the plurality of neural engines, wherein the second segment of the input data has a second size.

13. The method of claim 8, further comprising: The kernel generates the processed numerical value based on the multiplication and addition operations of the numbers on the shifted portion of the working unit. as well as The processed digital value is stored in at least one of the plurality of neural engines.

14. The method of claim 13, further comprising: A second number multiplication and addition operation is performed on the processed numerical value by at least one of the plurality of neural engines.

15. An electronic system, the system comprising: Data buffer circuit; and Multiple neural engines, each of which is configured as follows: The corresponding working unit is obtained from the data buffer circuit among a plurality of working units, wherein each corresponding working unit corresponds to a portion of the input data segment; Store the corresponding working unit; A portion of the working unit is shifted by the input buffer circuit; and The kernel performs digital multiplication and addition operations on the shifted portion of the corresponding working unit that has been shifted by the input buffer circuit.

16. The electronic system of claim 15, further comprising: A raster circuit is configured to instruct the input buffer circuit of the data buffer circuit to shift a portion of the corresponding working unit; as well as A neural task manager configured to send task information to the raster circuitry, the task information indicating at least how the input data is segmented into the plurality of working units.

17. The electronic system of claim 15, wherein the segment of the input data is a subset of: a slice of the input data, a slice of the input data that vertically divides the input data, and a segment that horizontally divides each slice of the slice.

18. The electronic system of claim 15, wherein the segment of said input data has a first size; and Each of the plurality of neural engines is further configured to: obtain a corresponding second working unit from the data buffer circuit corresponding to a second segment of the input data, wherein the second segment of the input data has a second size.

19. The electronic system of claim 15, wherein each of the plurality of neural engines is further configured to: The kernel generates processed numerical values ​​based on the multiplication and addition operations of the numbers on the shifted portion of the corresponding working unit; and The processed digital value is stored in the accumulator of the neural engine.

20. The electronic system of claim 19, wherein a second digital multiplication and addition operation is performed on the processed digital value.

Images