Method and apparatus for neural processor scheduling

Abstract

A method and apparatus for neural processor scheduling are disclosed. The method includes receiving data regarding a layer of a neural network, generating partitions to be assigned to cores by partitioning the data, generating tiles by partitioning the partitions, and scheduling an operation order of the tiles based on whether the data is shared among the cores.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0167681, filed on December 3, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for scheduling neural processors. Background Technology

[0003] When two or more independent neural processors aim to accelerate a neural network, the processors need to perform operations by partitioning feature maps or weights. In this case, redundant data exists between the neural processors, increasing memory bandwidth.

[0004] When different neural processors divide a neural network and execute the corresponding partitions, frequent synchronization between neural processors is required to ensure operation and data copying, leading to performance degradation.

[0005] When considering only a single neural processor as in conventional methods, tile scheduling for hiding latency and data copying between kernels may not be taken into account. Therefore, a method is needed to appropriately partition a given neural network for two or more processors and schedule subnetworks within the neural processors. Summary of the Invention

[0006] This summary is provided to introduce, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0007] In one general aspect, a method for scheduling neural network operations includes: receiving data about the layers of the neural network; generating partitions to be allocated to kernels by dividing the data; generating blocks by dividing the partitions; and scheduling the operation order of the blocks based on whether the data is shared between kernels.

[0008] The data may include the input feature map of the layer or the weights of the layer.

[0009] The steps for generating partitions may include: generating partitions based on the partitioning strategy of the layers preceding the layer or the size of the data.

[0010] The step of generating partitions based on the partitioning strategy of the previous layer or the size of the data may include: generating partitions based on the partitioning strategy of the previous layer in response to the output of the previous layer being forwardable; and generating partitions by comparing the size of the input feature map of the layer with the size of the weights of the layer in response to the output of the previous layer being non-forwardable.

[0011] In response to the output of a previous layer being non-forwardable, the step of generating a partition by comparing the size of the input feature map of the layer with the size of the weights of the layer may include: in response to the input feature map or weights not being uniformly divided and allocated to the kernel, generating a partition by comparing the loss caused by memory size with the loss caused by imbalance.

[0012] The steps of generating blocks may include: generating blocks based on the partitioning strategy of the layers preceding the layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data.

[0013] The step of generating a block based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data may include: generating a block based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, and the block partitioning strategy of the previous layer in response to the output of the previous layer being forwardable; and generating a block by comparing the size of the input feature map of the layer with the size of the weights of the layer in response to the output of the previous layer being non-forwardable.

[0014] The steps to generate partitions may include dividing the data in the height or width direction.

[0015] The scheduling steps may include changing the order of block operations based on whether the results of block operations are shared between cores.

[0016] The step of changing the operation order of blocks based on whether the operation results of blocks are shared between cores may include: changing the operation order in response to the fact that the operation results of a block are shared between the core to which the block was allocated and other cores, so as to prioritize the operation of the block included in the block.

[0017] In another general aspect, an apparatus for scheduling operations of a neural network includes: a receiver configured to receive data about layers of the neural network; and a processor configured to: generate partitions to be allocated to cores by partitioning the data, generate blocks by partitioning the data, and schedule the operation order of the blocks based on whether the data is shared between cores.

[0018] The data may include the input feature map of the layer or the weights of the layer.

[0019] The processor can also be configured to generate partitions based on the partitioning strategy of the layers preceding the layer or the size of the data.

[0020] The processor can also be configured to: generate partitions based on the partitioning strategy of the previous layer in response to the output of the previous layer being forwardable; and generate partitions by comparing the size of the input feature map of the layer with the size of the weights of the layer in response to the output of the previous layer being non-forwardable.

[0021] The processor can also be configured to generate partitions by comparing the loss caused by memory size with the loss caused by imbalance in response to the input feature map or weights not being uniformly divided and allocated to the kernel.

[0022] The processor can also be configured to generate blocks based on the partitioning strategy of the layers preceding the layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data.

[0023] The processor can also be configured to: generate blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, and the block partitioning strategy of the previous layer in response to the output of the previous layer being forwardable; and generate blocks by comparing the size of the input feature map of the layer with the size of the weights of the layer in response to the output of the previous layer being non-forwardable.

[0024] The processor can also be configured to divide the data in the height or width direction.

[0025] The processor can also be configured to change the order of block operations based on whether the results of block operations are shared between cores.

[0026] The processor can also be configured to change the order of operations in response to the sharing of the operation result of a block between the core to which the block was allocated and other cores, so as to prioritize the operation of the block included in the block.

[0027] Other features and aspects will become clear from the following detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0028] Figure 1 An example of a neural network operation scheduling device is shown.

[0029] Figure 2 Show Figure 1 An example of a processor is shown in the image.

[0030] Figure 3 Examples of input and output data are shown.

[0031] Figure 4 Show Figure 1The diagram shows an example of the processor partitioning operation process.

[0032] Figure 5A This example shows how to partition the data and assign the partitioned data to the kernel.

[0033] Figure 5B An example of data partitioning is shown.

[0034] Figure 6 An example illustrating the process of partitioning data.

[0035] Figure 7A This shows an example of partitioning the kernel.

[0036] Figure 7B An example of partitioning is shown.

[0037] Figure 8 This example illustrates the process of partitioning.

[0038] Figure 9A This example shows a scheduling result where the inter-core data transfer latency is not hidden.

[0039] Figure 9B This example illustrates a scheduling result where the inter-core data transfer latency is hidden.

[0040] Figure 10 An example of a hardware implementation of a neural network operation scheduling device is shown.

[0041] Figure 11 Examples of broadcasting and synchronization are shown.

[0042] Figure 12 Show Figure 1 An example of the operation process of a neural network scheduling device.

[0043] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0044] The detailed structural or functional descriptions provided below are merely examples and are open to various changes and modifications. These examples are not to be construed as limited to the public disclosure, but should be understood to include all changes, equivalents, and substitutions within the scope of the disclosed ideas and techniques.

[0045] Components may be described herein using terms such as first, second, etc. Each of these terms is not used to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from one or more other components. For example, the first component may be referred to as the second component, and similarly, the second component may be referred to as the first component.

[0046] It should be noted that if the first component is described as "connected", "joined", or "engaged" to the second component, then although the first component may be directly connected, joined, or engaged to the second component, the third component may be "connected", "joined", or "engaged" between the first and second components.

[0047] Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It will also be understood that when the terms “comprising” and / or “including” are used herein, they indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0048] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Unless clearly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0049] In the following description, examples will be described in detail with reference to the accompanying drawings. When describing examples with reference to the accompanying drawings, the same reference numerals denote the same components, and repeated descriptions relating to the same components will be omitted.

[0050] Figure 1 An example of a neural network operation scheduling device (e.g., a device for scheduling neural processors) is shown.

[0051] Reference Figure 1 The neural network operation scheduling device 10 can perform scheduling for neural network operations. The neural network operation scheduling device 10 performs scheduling by dividing neural network operations and allocating the divided neural network operations to neural processors.

[0052] The neural network operation scheduling device 10 can allocate neural network operations to neural processors by receiving and dividing data about the layers of the neural network.

[0053] Data about the layers of a neural network can include the input feature maps or weights of the layers.

[0054] Neural networks (or artificial neural networks) can include statistical training algorithms that simulate biological neurons in machine learning and cognitive science. A neural network can represent a general model with problem-solving capabilities, where artificial neurons (nodes) that form a network through synaptic connections are trained to change the strength of synaptic connections.

[0055] A neural network may include layers. A layer may include nodes.

[0056] Neural networks can include deep neural networks (DNNs). Neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), perceptrons, feedforward networks (FFs), radial basis function networks (RBFs), deep feedforward networks (DFFs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), autoencoders (AEs), variational autoencoders (VAEs), denoising autoencoders (DAEs), sparse autoencoders (SAEs), Markov chains (MCs), Hopfield networks (HNs), Boltzmann machines (BMs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), deep convolutional networks (DCNs), deconvolutional networks (DNs), deep convolutional inverse graph networks (DCIGNs), generative adversarial networks (GANs), liquid state machines (LSMs), extreme learning machines (ELMs), echo state networks (ESNs), deep residual networks (DRNs), differentiable neural computers (DNCs), neural Turing machines (NTMs), capsule networks (CNs), Kohonen networks (KNs), and attention networks (ANs).

[0057] The neural network operation scheduling device 10 includes a receiver 100 and a processor 200. The neural network operation scheduling device 10 may also include a memory 300.

[0058] Receiver 100 can receive data about the layers of a neural network. Receiver 100 includes a receiving interface. Receiver 100 can receive data about the layers of the neural network from an external device or memory 300. Receiver 100 can output the received data to processor 200.

[0059] Processor 200 can process data stored in memory 300. Processor 200 can execute computer-readable code (e.g., software) and instructions triggered by processor 200 stored in memory 300.

[0060] "Processor 200" can be a data processing device implemented in hardware, comprising circuitry with a physical structure that performs the desired operation. For example, the desired operation may include instructions or code contained in a program.

[0061] For example, hardware-implemented data processing devices may include microprocessors, central processing units (CPUs), processor cores, multi-core processors, multiprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0062] Processor 200 may include a neural processor. Optionally, processor 200 may be implemented separately from the neural processor. The neural processor may include a core.

[0063] Processor 200 can generate partitions that will be allocated to kernels by dividing the data about the layers of the neural network. Processor 200 can generate partitions based on the partitioning strategy of previous layers or the size of the data.

[0064] Processor 200 can partition data in the height or width direction. Additionally, processor 200 can partition data based on channels or batches. In other words, processor 200 can generate partitions by partitioning data in the height, width, channel, or batch direction.

[0065] In response to the fact that the output of the previous layer is forwardable, the processor 200 can generate partitions based on the partitioning strategy of the previous layer.

[0066] Since the output of the previous layer is not forwardable, the processor 200 can generate partitions by comparing the size of the input feature map of the layer with the size of the weights of the layer.

[0067] In this example, in response to the input feature map or weights not being evenly divided and allocated to the kernel, the processor 200 can generate partitions by comparing the loss caused by memory size with the loss caused by imbalance.

[0068] Processor 200 can generate tiles by dividing the generated partitions. Processor 200 can generate tiles based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, the tile division strategy of the previous layer, or the size of the data.

[0069] In response to the forwardable output of the previous layer, the processor 200 can generate blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, and the block partitioning strategy of the previous layer.

[0070] Since the output of the previous layer is not forwardable, the processor 200 can generate blocks by comparing the size of the input feature map of the layer with the size of the weights of the layer.

[0071] The processor 200 can generate blocks by partitioning the data in the height or width direction.

[0072] Processor 200 can schedule the operation order of blocks based on whether data is shared between cores. Processor 200 can change the operation order of blocks based on whether the operation results of blocks are shared. In response to the operation result of a block being shared between the core to which the block was allocated and other cores, processor 200 can change the operation order to prioritize the operation of that block included in the block.

[0073] The memory 300 may store instructions (or programs) that can be executed by the processor 200. For example, the instructions may include instructions for performing operations of the processor 200 and / or operations of each element of the processor 200.

[0074] The memory 300 is implemented as a volatile memory device or a non-volatile memory device.

[0075] Volatile memory devices can be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero-capacitor RAM (Z-RAM), or dual-transistor RAM (TTRAM).

[0076] Non-volatile memory devices can be implemented as electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic RAM (MRAM), spin-transfer torque (STT)-MRAM, conductive bridged RAM (CBRAM), ferroelectric RAM (FeRAM), phase change RAM (PRAM), resistive RAM (RRAM), nanotube RRAM, polymer RAM (PoRAM), nanofloating gate memory (NFGM), holographic memory, molecular electronic memory devices, or insulation resistance variation memory.

[0077] Figure 2 Show Figure 1 An example of a processor is shown in the image.

[0078] Reference Figure 2 The processor 200 may include an inter-core partitioning policy determiner 210, an intra-core partitioning policy determiner 230, and a scheduler 250. The processor 200 may also include a synchronizer 270 and a broadcaster 290.

[0079] The inter-core partitioning strategy determiner 210 generates partitions to be allocated to cores by partitioning data about the layers of a neural network. The inter-core partitioning strategy determiner 210 determines an inter-core partitioning strategy for partitioning the data and partitions the data according to the determined strategy.

[0080] Inter-kernel partitioning strategies can include information about the method of partitioning data for layers of a neural network. For example, an inter-kernel partitioning strategy can include the direction of data partitioning, the number of partitions to be generated, and the location where the data partitioning is performed. (See also...) Figures 5A to 6 The operation of the inter-core partitioning strategy determiner 210 is described in detail.

[0081] The kernel partitioning strategy determiner 230 can generate blocks by dividing the data into partitions. The kernel partitioning strategy determiner 230 can determine the kernel partitioning strategy used to divide the partitions and divide the partitions according to the determined strategy.

[0082] The kernel partitioning strategy may include information about the method of partitioning. For example, the kernel partitioning strategy may include the direction of partitioning, the number of blocks to be generated by the partitioning, and the location where the partitioning is performed. (See also...) Figures 7A to 8 The operation of the kernel partitioning strategy determiner 230 is described in detail.

[0083] Scheduler 250 can schedule the operation order of partitioned blocks. Scheduler 250 can schedule the operation order of blocks based on whether data is shared between cores.

[0084] Scheduler 250 can perform scheduling by allocating partitioned blocks to cores and changing the order of operations on the blocks.

[0085] Synchronizer 270 can perform synchronization between cores. Synchronizer 270 can correct synchronization requests received from broadcaster 290 from the cores and synchronization between cores.

[0086] Broadcaster 290 can receive data from memory 300 and broadcast the received data to the core. Broadcaster 290 can send the received data to the core via a single direct memory access (DMA).

[0087] Reference Figure 10 and Figure 11 The operation of synchronizer 270 and broadcaster 290 are described in detail.

[0088] Figure 3 Examples of input and output data are shown. Figure 4 Show Figure 1 The diagram shows an example of the processor partitioning operation process.

[0089] Reference Figure 3 and Figure 4 Data about the layers of a neural network can include the layer's input feature map (IFM) and weights. Figure 3 An example of a feature map is shown.

[0090] Feature maps 310 and 330 may include channels, each of which may be in the form of a two-dimensional (2D) matrix. That is, feature maps 310 and 330 may have height and width.

[0091] For example, feature map 310 may have 32 channels, a height of 149, and a width of 149, while feature map 330 may have 32 channels, a height of 147, and a width of 147. The structure of a feature map can be represented as (batch, channel, height, width).

[0092] In this example, the output feature map (OFM) after feature map 310 passes through the layer can have 32 channels and a width and height of 147. Additionally, the output of the layer for feature map 330 can have 64 channels, a height of 147, and a width of 147.

[0093] The kernel (or weight filter) of the layer used to process feature maps 310 and 330 can be in the form of a 3×3 matrix. Figure 3 In the above, conv2_3x3_s1 can represent a convolutional neural network that performs two-dimensional matrix convolution operations, has a kernel in the form of a 3×3 matrix, and a stride of 1; conv3_3x3_s1 can represent a convolutional neural network that performs three-dimensional matrix convolution operations, has a kernel in the form of a 3×3 matrix, and a stride of 1.

[0094] The inter-core partitioning strategy determiner 210 can generate partitions by dividing the input feature map or weights. The inter-core partitioning strategy determiner 210 can receive the Nth layer (or the data of the Nth layer) and determine the partitioning strategy for the data of the Nth layer.

[0095] The goals of a partitioning strategy can include ensuring forwardability, load balancing, and bandwidth. Forwardability can mean reusing the IFM of subsequent layers while storing the OFM of the previous layer in local memory, rather than moving the OFM of the previous layer from local memory (e.g., SRAM) to global memory (e.g., DRAM).

[0096] Processor 200 may consider forwardability to determine inter-core partitioning and intra-core partitioning strategies to reduce latency and required bandwidth.

[0097] Load balancing can mean distributing tasks (e.g., operations) evenly across the cores.

[0098] Bandwidth may include memory bandwidth. Memory bandwidth may represent the rate at which a neural processor (e.g., a core) reads data from memory 300 or stores data in memory 300.

[0099] Inter-core partitioning policy determiner 210 determines the partitioning policy for data to ensure forwardability, load balancing, and bandwidth. Inter-core partitioning policy determiner 210 can use the (N-1)th partitioning policy to determine the partitioning policy for layer N. Inter-core partitioning policy determiner 210 generates partitions by dividing the data about layer N based on the determined partitioning policy.

[0100] Reference Figures 5A to 6 The data partitioning operation of the inter-core partitioning strategy determiner 210 is described in detail.

[0101] The kernel partitioning strategy determiner 230 can determine the kernel partitioning strategy. The kernel partitioning strategy determiner 230 can generate blocks by dividing the generated partitions based on the determined kernel partitioning strategy.

[0102] The goals of an intra-core partitioning strategy can include hiding halo regions and ensuring bandwidth. A halo region can represent the area or transmission time where data is sent from one core to another to perform operations at subsequent layers.

[0103] The intra-core partitioning strategy determiner 230 can determine the intra-core partitioning strategy used to divide the partitions, in order to hide the halo region and ensure bandwidth. The intra-core partitioning strategy determiner 230 can use the intra-core partitioning strategy of layer (N-1) and the inter-core partitioning strategy of layer N to divide the partitions of layer N.

[0104] Reference Figures 7A to 8 Describe in detail the operation of the kernel partitioning strategy determiner 230.

[0105] In the following text, reference will be made to Figures 5A to 6 The operation of the inter-core partitioning strategy determiner 210 is described in detail.

[0106] Figure 5A This example shows how to partition the data and assign the partitioned data to the kernel. Figure 5B An example of data partitioning is shown. Figure 6 An example illustrating the process of partitioning data.

[0107] Reference Figure 5A , Figure 5B and Figure 6 The inter-core partitioning strategy determiner 210 can generate partitions by dividing the data about the layers of the neural network. In this case, partitions can be assigned to cores 510, 530, and 550.

[0108] One core can correspond to one neural processor. A neural processor may include a core and SRAM. (See reference...) Figure 10 Provide a detailed description of an example of a hardware implementation of a neural processor.

[0109] The inter-core partitioning strategy determiner 210 can generate partitions 571, 572, and 573 from the input feature map of the partitioning layer. Furthermore, the inter-core partitioning strategy determiner 210 can generate partitions from the weights 591, 592, and 593 of the partitioning layer.

[0110] The inter-core partitioning strategy determiner 210 can generate partitions 571, 572, and 573 by partitioning the input feature map in the height, width, channel, or batch directions. Although in Figure 5A and Figure 5B In the example, the input feature map is partitioned in the height direction, but in some examples, the input feature map can be partitioned in the width direction, channel direction, or batch direction.

[0111] The inter-core partitioning strategy determiner 210 generates partitions by dividing weights 591, 592, and 593 to allocate the divided weights 591, 592, and 593 to different cores. When weights 591, 592, and 593 are divided, a piece of input data can be sent to all cores 510, 530, and 550.

[0112] Despite Figure 5A In the example, weights 591, 592, and 593 are assigned based on the corresponding kernel. However, in some examples, weights 591, 592, and 593 can be assigned in the height, width, channel, or batch directions. Figure 5A In this context, *.IFM_H indicates that the input feature map is divided in the height direction, and *.OFM_C indicates that the output feature map is divided in the channel direction.

[0113] Specifically, in operation 610, the inter-core partitioning policy determiner 210 can determine whether the (N-1)th layer is forwardable. That is, the inter-core partitioning policy determiner 210 can determine whether the operation of the current layer is executable while maintaining the partitioning policy and scheduling method of the previous layer.

[0114] In response to the (N-1)th layer being forwardable, in operation 611, the inter-core partitioning policy determiner 210 can determine whether the inter-core partitioning policy of the (N-1)th layer will partition the input feature map. In operation 612, in response to the inter-core partitioning policy of the (N-1)th layer partitioning the input feature map, the inter-core partitioning policy determiner 210 can partition the input feature map of the Nth layer.

[0115] In operation 613, in response to the inter-core partitioning policy of layer (N-1) not partitioning the input feature map, the inter-core partitioning policy determiner 210 can determine whether the inter-core partitioning policy of layer (N-1) will partition the weights. In operation 614, in response to the inter-core partitioning policy of layer (N-1) partitioning the weights, the inter-core partitioning policy determiner 210 can generate partitions by partitioning the weights of layer N.

[0116] In operation 630, in response to the (N-1)th layer being non-forwardable, the inter-core partitioning policy determiner 210 can compare the size (or dimensions) of the input feature map with the size (or dimensions) of the weights. In this case, the size can be the memory size.

[0117] In operation 631, in response to the input feature map being larger than the weights, the inter-kernel partitioning strategy determiner 210 can determine whether the input feature map can be uniformly partitioned and allocated to all kernels.

[0118] In response to determining that the input feature map can be uniformly partitioned and allocated to all cores, in operation 632, the inter-core partitioning policy determiner 210 partitions the input feature map. In operation 633, in response to determining that the input feature map cannot be uniformly partitioned and allocated to all cores, the inter-core partitioning policy determiner 210 compares the loss caused by memory size with the loss caused by imbalance.

[0119] In response to the loss caused by memory size being greater than the loss caused by imbalance, in operation 632, the inter-core partitioning policy determiner 210 can partition the input feature map. In operation 634, in response to the loss caused by memory size being no greater than the loss caused by imbalance, the inter-core partitioning policy determiner 210 can partition the weights.

[0120] In operation 650, in response to the fact that the size of the input feature map is no greater than the size of the weights, the inter-core partitioning strategy determiner 210 can determine whether the weights can be partitioned evenly and distributed to all cores.

[0121] In response to determining that the weights can be evenly partitioned and allocated to all cores, in operation 651, the inter-core partitioning policy determiner 210 may partition the weights. In operation 652, in response to determining that the weights cannot be evenly partitioned and allocated to all cores, the inter-core partitioning policy determiner 210 may compare the loss caused by memory size with the loss caused by imbalance.

[0122] In this case, in response to the loss caused by memory size being greater than the loss caused by imbalance, in operation 651, the inter-core partitioning policy determiner 210 can partition the weights. In operation 653, in response to the loss caused by memory size being no greater than the loss caused by imbalance, the inter-core partitioning policy determiner 210 can partition the input feature map.

[0123] The inter-core partitioning strategy determiner 210 can consider, with the highest priority, whether the feature map of the previous layer is forwardable, and if it is not forwardable, determine the partitioning strategy to minimize the required bandwidth.

[0124] Regarding the operational characteristics of the operator, considering whether the amount of operation can be allocated to each core as equally as possible, the inter-core partitioning strategy determiner 210 can determine a partitioning strategy to maximize the efficiency of the operator.

[0125] In the following text, reference will be made to Figure 7A , Figure 7B and Figure 8 Describe in detail the operation of the kernel partitioning strategy determiner 230.

[0126] Figure 7A This shows an example of partitioning the kernel. Figure 7B An example of partitioning is shown. Figure 8 This example illustrates the process of partitioning.

[0127] Reference Figures 7A to 8 The kernel partitioning strategy determiner 230 can generate blocks 710 to 770 by dividing the kernel into partitions. The kernel partitioning strategy determiner 230 can generate blocks 710, 720, 730, 740, 750, 760 and 770 by dividing the partitions allocated to the kernel.

[0128] The kernel partitioning strategy determiner 230 can generate blocks 710 to 770 by partitioning in the height direction, width direction, channel direction, or batch direction.

[0129] Figure 7A and Figure 7B Blocks 710 to 770 shown can be examples of partitioned blocks. Although blocks 710 to 770 are generated by dividing the blocks into partitions in the vertical direction, in some examples, blocks 710 to 770 can be generated by dividing the blocks into partitions in different directions.

[0130] The kernel partitioning strategy determiner 230 can generate blocks based on the partitioning strategy of the layers preceding the current layer, the partitioning strategy of the current layer, the block partitioning strategy of the previous layer, or the size of the data.

[0131] Since the output of the previous layer is forwardable, the kernel partitioning strategy determiner 230 can generate blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, and the block partitioning strategy of the previous layer.

[0132] Since the output of the previous layer is not forwardable, the kernel partitioning strategy determiner 230 can generate blocks by comparing the size of the input feature map of the layer with the size of the weights of the layer.

[0133] Specifically, in operation 810, the kernel partitioning policy determiner 230 can determine whether the (N-1)th layer is forwardable. That is, the kernel partitioning policy determiner 230 can determine whether the operation of the current layer is executable while maintaining the partitioning policy and scheduling method of the previous layer.

[0134] In operation 811, in response to the (N-1)th layer being forwardable, the intra-core partitioning policy determiner 230 can determine whether the inter-core partitioning policy and intra-core partitioning policy of the (N-1)th layer will partition the input feature map.

[0135] In operation 812, in response to the inter-core partitioning strategy and intra-core partitioning strategy of the (N-1)th layer, the input feature map will be partitioned, and the intra-core partitioning strategy determiner 230 can determine whether the inter-core partitioning strategy of the Nth layer will partition the input feature map.

[0136] In response to the inter-core partitioning strategy of the Nth layer, the input feature map will be divided. In operation 813, the intra-core partitioning strategy determiner 230 can generate blocks by dividing the input feature map of the Nth layer.

[0137] In operation 814, in response to the inter-kernel partitioning strategy of the Nth layer not partitioning the input feature map, the intra-kernel partitioning strategy determiner 230 can compare the size of the input feature map with the size of the weights.

[0138] In operation 815, in response to the input feature map size not being greater than the weight size, the kernel partitioning strategy determiner 230 can generate blocks by dividing the weights. In response to the input feature map size being greater than the weight size, in operation 813, the kernel partitioning strategy determiner 230 can divide the input feature map.

[0139] In operation 816, in response to the fact that the inter-core partitioning strategy and intra-core partitioning strategy of the (N-1)th layer do not partition the input feature map, the intra-core partitioning strategy determiner 230 can determine whether the inter-core partitioning strategy and intra-core partitioning strategy of the (N-1)th layer will partition the weights.

[0140] In operation 817, in response to the inter-core partitioning policy and intra-core partitioning policy of layer (N-1), weights will be assigned, and the intra-core partitioning policy determiner 230 can determine whether the inter-core partitioning policy of layer N will assign weights.

[0141] In response to the inter-core partitioning strategy of the Nth layer, weights will be assigned. In operation 818, the intra-core partitioning strategy determiner 230 can generate blocks by assigning weights.

[0142] In operation 819, in response to the inter-kernel partitioning strategy of the Nth layer not dividing the weights, the intra-kernel partitioning strategy determiner 230 can compare the size of the input feature map with the size of the weights.

[0143] In operation 820, in response to the input feature map being larger than the weights, the kernel partitioning policy determiner 230 partitions the input feature map. In response to the input feature map being no larger than the weights, in operation 818, the kernel partitioning policy determiner 230 partitions the weights.

[0144] Taking into account the inter-core partitioning strategy / intra-core partitioning strategy of the previous layer and the inter-core partitioning strategy of the current layer, the intra-core partitioning strategy determiner 230 can determine a method for dividing or scheduling partitions to hide the inter-core direct memory access (DMA) latency.

[0145] In response to the fact that the previous layer is not forwardable, the kernel partitioning policy determiner 230 can generate blocks 710 to 770 by partitioning the bandwidth with the highest priority.

[0146] Figure 9A An example of scheduling results where inter-core data transfer latency is not hidden is shown. Figure 9B This example illustrates a scheduling result where the inter-core data transfer latency is hidden.

[0147] Scheduler 250 can schedule the operation order of blocks 910, 920, 930, 940, 950, 960, and 970 based on whether data is shared between cores. Scheduler 250 can change the operation order of blocks 910 to 970 based on whether the operation results of blocks 910 to 970 are shared.

[0148] In response to the fact that the operation result of a block is shared between the core to which the block was allocated and other cores, the scheduler 250 may change the operation order so as to prioritize the operation of the block included in the block.

[0149] Figure 9A This shows the schedule where the operation order of blocks 910 to 970 is not changed by scheduler 250. Figure 9B This shows the scheduling after the order of operations in blocks 910 to 970 was changed by scheduler 250.

[0150] Scheduler 250 can maximize the reusability of data broadcast to the neural processor and schedule the operation order of blocks 910 to 970 if feature map data is shared with adjacent layers in order to hide the inter-core data transfer latency.

[0151] When performing neural network operations using kernels, data sharing between kernels may be required. For example, the result of an operation performed by kernel #0 might be used for an operation performed by kernel #1. The halo region can represent the inter-kernel data transfer delay.

[0152] Scheduler 250 can hide the halo region and DMA time by prioritizing operations on blocks that include the halo region and then performing operations on other blocks.

[0153] In other words, scheduler 250 can improve core utilization by changing the operation order of blocks 910 to 970 to perform operations on other blocks by cores while transmission delays occur, so that operations on blocks that require inter-core data sharing are prioritized.

[0154] exist Figure 9A In the example, when scheduler 250 does not perform individual scheduling, idle time may occur for halo transmission, reducing core utilization. Scheduler 250 can change the operation order of blocks 940 and 950, thereby hiding the halo region while performing operations on other blocks.

[0155] Through the scheduling process described above, the neural network operation scheduling device 10 can improve the utilization of all cores.

[0156] Figure 10 This illustrates an example of a hardware implementation of a neural network operation scheduling device. Figure 11 Examples of broadcasting and synchronization are shown.

[0157] Reference Figure 10 and Figure 11 The neural network operation scheduling device 10 may include a synchronizer 270 and a broadcaster 290. The neural network operation scheduling device 10 may also include a neural processor (NPU) 1010, a control core 1030, a DMA engine 1050, and a global memory 1070. For example, the global memory 1070 may include DRAM. Figure 10 In this system, synchronizer 270, broadcaster 290, NPU 1010, control core 1030 and DMA engine 1050 may be included in the NPU subsystem of neural network operation scheduling device 10.

[0158] The neural processor 1010 may each include a tensor kernel and local memory. For example, the local memory may include SRAM.

[0159] Synchronizer 270 and broadcaster 290 can be implemented inside the processor or as separate hardware modules.

[0160] Synchronizer 270 performs synchronization between cores. Synchronizer 270 ensures synchronization between cores as well as synchronization between requests received from cores for broadcast data. Synchronizer 270 reduces the number and cost of synchronization between cores.

[0161] For example, when the same data is copied once to the local memory (e.g., SRAM) of each core via broadcaster 290, synchronizer 270 can synchronize the operation of each core.

[0162] Synchronizer 270 can resolve the problem that differences in operation time between cores cause data to be copied to memory regions in use and affect correctness, or conversely, it can resolve the problem that frequent synchronization reduces core utilization.

[0163] Synchronizer 270 can mark the hardware at the point when each core is ready to load data so that the necessary operations can be performed again, and actually perform data copying when all cores are marked, thereby reducing the number of synchronizations while ensuring the exquisiteness of the operation, thus reducing synchronization costs.

[0164] Broadcaster 290 can broadcast data in DRAM 1070 to each of neural processors 1010. For example, broadcaster 290 can use a single DMA to send data from DRAM to SRAM. Thus, broadcaster 290 can reduce the bandwidth required for the system bus.

[0165] When partitioning data and operations along the height direction of the input feature map, the weights may be redundant and are therefore copied to the local memory of each core. Similarly, when partitioning along the channel direction of the output feature map, the input feature map may be redundant and is therefore copied. This copying of redundant data can significantly impact the required bandwidth of the system bus.

[0166] The broadcaster 290 can perform a data copy only once via the system bus and copy the copied data to the local memory of each core, thereby preventing an increase in the required bandwidth due to redundant data movement.

[0167] Figure 12 Show Figure 1 An example of the operational flow of a neural network operation scheduling device (e.g., a method for scheduling neural processors).

[0168] In operation 1210, receiver 100 may receive data about the layers of a neural network. The data about the layers of the neural network may include the input feature maps or weights of the layers.

[0169] In operation 1230, processor 200 can generate partitions to be allocated to cores by dividing the data about the layer. Processor 200 can generate partitions based on the partitioning strategy of the layers preceding the layer or the size of the data.

[0170] If the output of a previous layer is forwardable, the processor 200 can generate partitions based on the partitioning strategy of the previous layer. If the output of a previous layer is not forwardable, the processor 200 can generate partitions by comparing the size of the input feature map of the layer with the size of the layer's weights.

[0171] In response to the input feature map or weights not being evenly divided and allocated to the kernel, the processor 200 can generate partitions by comparing the loss caused by memory size with the loss caused by imbalance.

[0172] In operation 1250, processor 200 can generate blocks by partitioning. Processor 200 can generate blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data.

[0173] In response to the forwardable output of the previous layer, the processor 200 can generate blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, and the block partitioning strategy of the previous layer.

[0174] Since the output of the previous layer is not forwardable, the processor 200 can generate blocks by comparing the size of the input feature map of the layer with the size of the weights of the layer.

[0175] The processor 200 can divide the data in the height or width direction.

[0176] In operation 1270, processor 200 may schedule the operation order of blocks based on whether data is shared between cores. Processor 200 may change the operation order of blocks based on whether the operation results of blocks are shared. In response to the operation result of a block being shared between the core to which the block was allocated and other cores, processor 200 may change the operation order to prioritize the operation of that block included in the block.

[0177] The units described herein may be implemented using hardware components, software components, and / or combinations thereof. The processing device may be implemented using one or more general-purpose or special-purpose computers (such as processors, controllers and arithmetic logic units (ALUs), DSPs, microcomputers, FPGAs, programmable logic units (PLUs), microprocessors, or any other means capable of responding to and executing instructions in a defined manner). The processing device may run an operating system (OS) and one or more software applications running on the OS. The processing device may also access, store, manipulate, process, and create data in response to the execution of the software. For the purpose of brevity, the description of the processing device is used as the singular; however, those skilled in the art will understand that the processing device may include multiple processing elements and various types of processing elements. For example, the processing device may include multiple processors or a single processor and a single controller. Furthermore, different processing configurations are feasible (such as parallel processors).

[0178] Software may include computer programs, code segments, instructions, or combinations thereof, to independently or consistently instruct and / or configure a processing device to operate as desired. Software and data may be permanently or temporarily implemented in any type of machine, component, physical or virtual equipment, computer storage medium, or device, or permanently or temporarily implemented in propagating signal waves capable of providing instructions or data to the processing device or interpretable by the processing device. Software may also be distributed across networked computer systems, enabling it to be stored and executed in a distributed manner. Software and data may be stored on one or more non-transitory computer-readable recording media.

[0179] The methods described above can be recorded in a non-transitory computer-readable medium, which includes program instructions for implementing the various operations of the examples described above. The medium may also include data files, data structures, etc., alone or in combination with the program instructions. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the examples, or they may be of types known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical media (such as CD-ROMs, DVDs, and / or Blu-ray discs), magneto-optical media (such as optical discs), and hardware devices specifically configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.). Examples of program instructions include both machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0180] The apparatus described above can be configured to act as one or more software modules to perform the operations of the examples described above, or vice versa.

[0181] Several examples have been described above. However, it should be understood that various modifications can be made to these examples. For example, suitable results may be achieved if the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0182] Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method for scheduling a neural processor, comprising: Receive data about the layers of the neural network; The data is divided to create partitions that will be allocated to the core; Blocks are generated by dividing the area into partitions; and The order of block operations is determined by whether data is shared between cores. The steps for generating partitions include: determining a partitioning strategy for dividing data based on forwardability, and generating partitions by dividing data based on the determined partitioning strategy.

2. The method according to claim 1, wherein, The data includes the input feature map of the layer or the weights of the layer.

3. The method according to claim 1, wherein, The steps of generating partitions by dividing data based on a determined partitioning strategy for dividing the data include: generating partitions based on the partitioning strategy of a previous layer or the size of the data, wherein the previous layer is the layer preceding the previous layer.

4. The method according to claim 3, wherein, The steps for generating partitions based on the partitioning strategy of the previous layer or the size of the data include: The response is that the output of the previous layer is forwardable, and partitions are generated based on the partitioning strategy of the previous layer; and Since the output of the previous layer is not forwardable, partitions are generated by comparing the size of the input feature map of the layer with the size of the weights of the layer.

5. The method according to claim 4, wherein, The step of generating a partition by comparing the size of the input feature map of the layer with the size of the weights of the layer in response to the output of the previous layer being non-forwardable includes: generating a partition by comparing the loss caused by memory size with the loss caused by imbalance in response to the input feature map or weights not being uniformly divided and allocated to the kernel.

6. The method according to claim 1, wherein, The steps for generating blocks include: generating blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data, wherein the previous layer is the layer preceding the layer.

7. The method according to claim 6, wherein, The steps of generating blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data include: In response to the forwardable nature of the output from the previous layer, blocks are generated based on the partitioning strategy of the previous layer, the partitioning strategy of the layer in question, and the block partitioning strategy of the previous layer; and Since the output of the previous layer is not forwardable, blocks are generated by comparing the size of the input feature map of the layer with the size of the weights of the layer.

8. The method according to any one of claims 1 to 7, wherein, The steps to generate partitions include dividing the data in either the height or width direction.

9. The method according to claim 1, wherein, The scheduling steps include changing the order of block operations based on whether the results of block operations are shared among cores.

10. The method according to claim 9, wherein, The step of changing the operation order of blocks based on whether the operation results of blocks are shared between cores includes: in response to the operation results of a block being shared between the core to which the block was allocated and other cores, changing the operation order so as to preferentially operate the block included in the generated block.

11. A device for scheduling a neural processor, comprising: The receiver is configured to receive data about the layers of the neural network; and The processor is configured as follows: The data is divided to create partitions that will be allocated to the core. Blocks are generated by dividing the space into partitions, and The order of block operations is determined by whether data is shared between cores. The processor is configured as follows: The partitioning strategy for dividing the data is determined based on forwardability, and partitions are generated by dividing the data based on the determined partitioning strategy.

12. The device according to claim 11, wherein, The data includes the input feature map of the layer or the weights of the layer.

13. The device according to claim 11, wherein, The processor is also configured to generate partitions based on the partitioning strategy of a previous layer or the size of the data, where the previous layer is the layer preceding the previous layer.

14. The device according to claim 13, wherein, The processor is also configured as follows: The response is that the output of the previous layer is forwardable, and partitions are generated based on the partitioning strategy of the previous layer; and Since the output of the previous layer is not forwardable, partitions are generated by comparing the size of the input feature map of the layer with the size of the weights of the layer.

15. The device according to claim 14, wherein, The processor is also configured to generate partitions by comparing the loss caused by memory size with the loss caused by imbalance in response to the input feature map or weights not being uniformly divided and allocated to the core.

16. The device according to claim 11, wherein, The processor is also configured to generate blocks based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, the block partitioning strategy of the previous layer, or the size of the data, wherein the previous layer is the layer preceding the layer.

17. The device according to claim 16, wherein, The processor is also configured as follows: In response to the output of the previous layer being forwardable, blocks are generated based on the partitioning strategy of the previous layer, the partitioning strategy of the layer, and the block partitioning strategy of the previous layer; and Since the output of the previous layer is not forwardable, blocks are generated by comparing the size of the input feature map of the layer with the size of the weights of the layer.

18. The device according to any one of claims 11 to 17, wherein, The processor is also configured to divide the data in either the height or width direction.

19. The device according to claim 11, wherein, The processor is also configured to change the order of block operations based on whether the results of block operations are shared between cores.

20. The device according to claim 19, wherein, The processor is also configured to: in response to the sharing of the operation result of a block between the core to which the block was allocated and other cores, change the operation order so as to preferentially operate the block included in the generated block.

Images