Dynamically accurate data representation in neural network kernels

Abstract

A system for neural network computation is provided. A neural network processor includes a plurality of neural cores. The neural network processor has one or more processor precisions per activation. The processor is configured to accept data having a processor feature dimension. A conversion circuit is coupled to the neural network processor and is adapted to: receive an input data tensor having per-channel input precision on one or more features; convert the input data tensor from the input precision to the processor precision; divide the input data into a plurality of blocks, each block conforming to one of the processor feature dimensions; and provide each of the plurality of blocks to one of the plurality of neural cores. The neural network processor is adapted to compute outputs of one or more neural network layers through the plurality of neural cores.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to multi-core hardware implementations of neural networks, and more particularly, to data representation for dynamic precision in neural network kernels. Summary of the invention

[0002] According to an embodiment of the present disclosure, a method and a computer program product for operating a neural network are provided. In various embodiments, an input data tensor is received at a neural network processor including a plurality of neural cores. The input data tensor has a feature dimension at an input bit precision. The neural network processor is configured for one or more processor feature dimensions at one or more processor bit precisions. The input data is converted from the input bit precision to one of the processor bit precisions. The input data tensor is divided into a plurality of blocks, each block conforming to one of the processor feature dimensions. Each of the plurality of blocks is provided to one of the plurality of neural cores. The plurality of neural cores calculate the output of one or more neural network layers.

[0003] According to an embodiment of the present disclosure, a system for neural network computing is provided. A neural network processor includes a plurality of neural cores. The neural network processor has one or more processor precisions per activation. The processor is configured to accept data having a processor feature dimension. A conversion circuit is coupled to the neural network processor and is adapted to: receive an input data tensor having per-channel input precision on one or more features; convert the input data tensor from the input precision to the processor precision; divide the input data into a plurality of blocks, each block conforming to one of the processor feature dimensions; and provide each of the plurality of blocks to one of the plurality of neural cores. The neural network processor is adapted to calculate the output of one or more neural network layers through the plurality of neural cores. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A neural core according to an embodiment of the present disclosure is shown.

[0005] Figure 2 An exemplary inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0006] Figure 3 A multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0007] Figure 4 A conversion according to an embodiment of the present disclosure is shown.

[0008] Figure 5 A method of operating a neural network according to an embodiment of the present disclosure is shown.

[0009] Figure 6 A computing node according to an embodiment of the disclosure is depicted. DETAILED DESCRIPTION

[0010] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one neuron is the input of the other neuron. A weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of another neuron.

[0011] A neuron computes its output, called an activation, by applying a nonlinear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result obtained by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0012] A neural network is a collection of one or more neurons. Neural networks are usually divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, and generally perform similar functions. An input layer is a layer that receives input from a source external to the neural network. An output layer is a layer that sends output to a destination external to the neural network. All other layers are intermediate processing layers. A multi-layer neural network is a neural network with more than one layer. A deep neural network is a multi-layer neural network with many layers.

[0013] A tensor is a multidimensional array of values. A tensor block is a contiguous subarray of elements in a tensor.

[0014] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all the parameters that control the activation functions σ of the neurons in this layer. The weight tensor contains all the weights that connect the inputs to this layer. The input data tensor contains all the data consumed as input by this layer. The output data tensor contains all the data computed as output by this layer. The intermediate data tensor contains any data produced by this layer as intermediate computations, such as partial sums.

[0015] The data tensors (input, output, and intermediate) of a layer can be 3-dimensional, where the first two dimensions can be interpreted as encoding spatial locations and the third dimension can be interpreted as encoding different features. For example, when the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, while the third dimension encodes the color of each location. Each element of the input data tensor X can be connected to each neuron via a separate weight, so the weight tensor W usually has 6 dimensions, concatenating the 3 dimensions of the input data tensor (input row a, input column b, input feature c) and the 3 dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the 3 output data tensor dimensions with an additional dimension o that indexes the parameters of the activation function σ.

[0016] The elements of the output data tensor Y of a layer can be calculated according to Formula 1, where the neuron activation function σ is configured by the activation function parameter vector V[i,j,k,:], and the weighted sum Z[i,j,k] can be calculated according to Formula 2.

[0017] Y[i,j,k]=σ(V[i,j,k,:];Z[i,j,k])

[0018] Formula 1

[0019]

[0020]

[0021] To simplify the representation, the weighted sum in Formula 2 can be referred to as the output, which is equivalent to using a linear activation function Y[i,j,k]=σ(Z[i,j,k])=Z[i,j,k], and it should be understood that the same statement can be applied without loss of generality when different activation functions are used.

[0022] In various embodiments, the computation of the output data tensor as described above is broken down into smaller problems. Each problem can then be solved in parallel on one or more neural cores, or on one or more cores of a conventional multi-core system.

[0023] Convolutional layers exploit spatially local correlations in natural sensor data by enforcing local connectivity patterns between neurons in adjacent layers: each neuron receives connections from only a small portion of the input data tensor. The extent of this connectivity is called the neuron's receptive field. All neurons that compute the same output feature element have the same set of weights and activation function parameters (called filters) and have a local receptive field. The size of the filters is fixed regardless of the input size, so convolutional layers have far fewer free parameters and a correspondingly smaller memory footprint than fully connected layers with the same input and output dimensions.

[0024] For example, in a spatial convolutional layer, the receptive field of each filter covers only a portion of the height and width of the input data tensor, but extends throughout the feature depth. This architecture ensures that the filters respond most strongly to spatially local input patterns. The layer computes its output by convolving each filter across the width and height of the input data tensor, computing the dot product between the entries of the filter and the input data at each location to produce a two-dimensional activation map for each filter.

[0025] The feature maps of all filters are stacked to form the complete output volume of the convolutional layer. Therefore, each entry in the output data tensor can also be interpreted as the output of a neuron that looks at a small area in the input and shares the filter parameters with the neuron in the same output feature map.

[0026] It should be understood that the size of the output data tensor of a convolutional layer is a function of the size of the input data tensor, the size of the filter, and the stride with which the filter is applied to the input data tensor.

[0027] The stride parameters stride_a, stride_b, and stride_c define the distance between each filter application, with a value of 1 meaning that each filter application is shifted by one pixel along the relevant dimension of the input data tensor. Larger stride values ​​reduce the amount of computation by computing only a subset of the possible output pixels.

[0028] It will be further appreciated that in various convolutions, zero padding may be applied to the input image in order to change the output image size relative to the input size, for example to make those sizes equal. For simplicity of illustration, image padding may be omitted in various examples, but it will be apparent that padding may be included in various embodiments without departing from the scope of the present disclosure.

[0029] In a convolutional layer, the 6-dimensional weight tensor contains many repeated blocks because all elements of the same output feature share the same filter weights replicated at each output position. The shared filter weights can be described more compactly by a dense 4-dimensional filter tensor F, which contains all the filters that compute the output features of the layer and is indexed by the output feature dimension (output features k) and the 3 filter input dimensions (filter rows r, filter columns s, filter features t).

[0030]

[0031] In Equation 3, the range of the summation of r, s, t can be reduced for i, j, k coordinates near the edge of the output data tensor to accommodate boundary conditions, which are omitted here for simplicity. The accessor function α(i, r; A) computes the row coordinate a of the input data element multiplied by the filter weight F[k, r, s, t] to compute the output data element Y[i, j, k], where A is a set of parameters that parameterize the row coordinate computation, such as the stride parameter stride_a and the padding parameter pad_a. Similar accessor functions β(j, s; B) and γ(r, t; Γ) compute the column and feature coordinates b and c.

[0032] In various embodiments, the convolutions described above are broken down into smaller problems. Each problem can then be solved in parallel in one or more neuromorphic cores or in one or more cores of a conventional multi-core system.

[0033] In some embodiments, the input data tensor is decomposed into blocks along its row and column dimensions that extend over the entire length of its feature dimension. In such embodiments, the kernel receives the input data block and the entire filter tensor F. In order to correctly compute the convolution of pixels at the boundaries of the input data blocks, the input data blocks can be generated so that they overlap. Alternatively, the partial sums from multiple kernels can be added together to obtain the final result.

[0034] Neural networks are used for a variety of tasks, such as image classification. Low-precision and / or limited neural network hardware accelerators can use less bit precision to implement tasks with lower precision for neuron activations (values ​​sent between layers) and parameters (such as neuron-to-neuron weights) than conventional computers (such as CPUs and GPUs). This saves both silicon area and computing energy without sacrificing task performance (such as classification accuracy).

[0035] However, reduced precision substrates face challenges such as converting inputs (which may be high precision) to compatible low precision representations. Such low precision representations may require specific formats, such as input vectors instead of scalars. Additionally, hardware precision is fixed at fabrication time, but different neural networks and different layers within a single neural network may benefit from different levels of precision.

[0036] Reference now Figure 1 , shows a neural core according to an embodiment of the present disclosure. The neural core 100 is a splicable computing unit that calculates a block of output tensors. The neural core 100 has M inputs and N outputs. In various embodiments, M=N. In order to calculate the output tensor block, the neural core multiplies the M×1 input tensor block 101 with the M×N weight tensor block 102, and accumulates the products into a weighted sum, which is then stored in a 1×N intermediate tensor block 103. The O×N parameter tensor block contains O parameters that specify each of the N neuron activation functions applied to the intermediate tensor block 103 to produce the 1×N output tensor block 105.

[0037] Multiple neural cores can be tiled in a neural core array. In some embodiments, the array is two-dimensional.

[0038] A neural network model is a set of constants that together specify the entire computation performed by the neural network, including the connection graph between neurons and the weights and activation function parameters of each neuron. Training is the process of modifying the neural network model to perform the desired function. Inference is the process of applying the neural network to inputs to produce outputs without modifying the neural network model.

[0039] An IPU is a type of processor that performs neural network inference. A neural inference chip is a specific physical instance of an IPU.

[0040] refer to Figure 2 , shows an exemplary inference processing unit (IPU) according to an embodiment of the present disclosure. IPU 200 includes a memory 201 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be calculated. IPU 200 includes an activation memory 202, which may be transient. Activation memory 202 may be divided into input and output areas, and stores neuron activations for processing. IPU 200 includes a neural computing unit 203, which has loaded the neural network model from model memory 201. Before each calculation step, input activation is provided from activation memory 202. The output from neural computing unit 203 is written back to activation memory 202 for processing on the same or another neural computing unit.

[0041] In various embodiments, scheduler 204 is included in IPU 200. In such embodiments, all operations in the IPU are directed by the scheduler. As described below, a central and / or distributed scheduler may be provided in various embodiments. The global scheduler may be referred to as a chip microengine, while the local scheduler may be referred to as a core microengine or a local controller. In various embodiments, the scheduler includes one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0042] refer to Figure 3 , shows a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure. IPU 300 includes a model memory 301 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be calculated. IPU 300 includes an activation memory 302 that may be transient. Activation memory 302 may be divided into input and output areas and stores neuron activations for processing. In some embodiments, IPU 300 includes an instruction memory 307 for storing chip-level instructions.

[0043] IPU 300 includes multiple cores 303 in array 305. Each core 303 includes a neural computing unit 333, which has loaded the neural network model from model memory 301. Each core also includes a local activation memory 332. Before each computation step, input activations are provided from local activation memory 332. The output from the neural computing unit 333 is written back to the activation memory 332 for processing on the same or another neural computing unit.

[0044] In various embodiments, a global scheduler 304 is included in the IPU 300. In various embodiments, a local core controller 334 is included on each core 303. In such embodiments, the direction of operation is shared between the global scheduler (chip microengine) and the local core controller (core microengine). In some embodiments, a chip-level instruction memory 306 is provided for storing chip-level instructions for execution by the chip microengine 304. In some embodiments, the core 303 includes a core-level instruction memory 336 for storing core-level instructions executed by the core microengine 334.

[0045] At 311, computation instructions are loaded from instruction memory 301 to each core 303 by global scheduler 304. At 312, global scheduler 304 loads parameters (e.g., neural network / synaptic weights) from model memory 301 to each core 303. At 313, neural network activation data is loaded from data memory 302 to each core 303 by global scheduler 304. At 314, the cores in matrix 305 perform computations to generate output neuron activations. In particular, the computation includes applying input synaptic weights to input activations. It will be appreciated that various methods may be used to perform such computations, including in silicodendrites and vector multiplication units. At 315, the results from the computations are stored in data memory 302. These stages may be pipelined to provide efficient use of neural computation units on each core.

[0046] It should be understood that inputs and outputs can be transferred from local activation memory 332 to global activation memory 302 as required by a given neural network. Similarly, model memory and instruction memory can have both chip-level and core-level components. At the core level, at 341, computation instructions are loaded into local instruction memory 336. At 342, parameters (e.g., neural network / synaptic weights) are loaded into model memory 331. At 343, neural network activation data is loaded into data memory 332. At 344, computation unit 332 computes output activations or partial sums. At 345, the computation results are output from data memory 332. In order to provide efficient use of each core, these stages can be pipelined.

[0047] Reference now Figure 4 , shows conversions according to embodiments of the present disclosure. In various embodiments, conversions from high precision to low precision and / or from scalar to vector are provided. In particular, the conversion block can receive high precision data in various formats and convert it to a hardware compatible low precision representation. Similarly, the conversion block can receive various formats of scalar or vector data and convert it to a hardware compatible vector representation.

[0048] The input image (eg, 401) is typically 3 color channels (eg, RGB), each channel having 8 bits of precision (encoding 0-255). For example, if the hardware uses R bits to represent each activation channel (where R < 8), various methods can be used to convert to R bits of precision.

[0049] Each 8-bit value can be converted to an R-bit value by dropping the 8-R least significant bits (e.g., in cases where all 8 bits of precision are not needed).

[0050] - Each 8-bit value can be converted to 8 / R R-bit values ​​(eg, for R=2, 8 / 2=4 2-bit values ​​are used).

[0051] - Each 8-bit value can be converted to less than 8 / R R-bit values ​​(e.g., for R=2, 2 2-bit values ​​can be used, dropping the 4 least significant bits).

[0052] An input image of size W×H×C (e.g., 401) can be represented as a W×H C-element vector or a W×H×C scalar, where W is the input width, H is the input height, and C is the number of input channels or colors. For example, if the hardware uses a T-element vector as input, there are multiple ways to convert to a T-element vector.

[0053] If T>C, then each C-element vector can be directly mapped to a T-element vector with TC zero entries, resulting in W×H T-element vectors.

[0054] If T>C, up to T / C pixels can be packed into each T element vector. For example, if C=3 and T=24, 8 pixels can be packed into a 4×2 tile with a stride of 4 width and 2 height, which produces W / 4×H / 2 T element vectors.

[0055] If T>C, up to T / C pixels can be packed into each T-element vector in a way that is optimized for computation in the next layer. For example, if C=3 and T=24, and the next layer implements 8×8 convolutions with a stride of 4, the 8 pixels can be packed into, for example, 8×1 tiles with a stride of 4, resulting in W / 4×H T-element vectors. Now, the next layer only needs to implement 1×8 convolution on the resulting input, which optimizes hardware utilization.

[0056] If T < C, each C-element vector can be unpacked into C / T T-element vectors, resulting in W×H×C / T T-element vectors. For example, if C=4 and T=2, each 4-element vector can be unpacked into 2 2-element vectors, resulting in W×H×2 2-element vectors.

[0057] In the illustrative example of the conversion, the input image 401 is 256×256×3, and the underlying hardware uses 4-bit activations and vectors of 32 elements. The input image 401 is distributed across a 4×4 kernel array 402. To complete the convolution, each kernel requires a (32+3)×(32+3)×3 portion of the original image 401 for a 7×7×3 filter. Using multiple rows of scratchpad memory 402, the 1×7×3(×4) most significant bits are packed into a 21-element vector (4 bits per element) that fits into a 32-element flat vector 404. As shown, a column stride of 2 is used, resulting in some data duplication, but determining the data size for the convolution layer.

[0058] The result of this conversion is to turn a 7×7×3 filter into a 7×1×21 filter. The vector utilization is 21 / 32≈66%. The packed image tensors 405, 406 are 256×128×32, so each core (in the 4×4 array 402) receives a 32×16×32 portion of the tensor (tensor).

[0059] As shown, the input blocks may be double buffered, alternating between processing one tensor 405 while sending a previous tensor 406 to the kernel array 402 .

[0060] It will be appreciated that the example given above is only one possible configuration of the input data tensor. For example, with a sample size of 4×7×1, the vector utilization is 24.5 / 32≈77%.

[0061] In various embodiments, flexible precision conversion is provided. In the exemplary neural network, each neural network layer uses low-precision weights to calculate low-precision activations, each low-precision activation is represented by R bits, and each low-precision weight is represented by S bits. As described above, the neurons in the layer receive input X (with elements x) from one (or more) previous layers. i Each neuron is based on the input and a set of weights W (with elements w i ) computes its state Y, these weights W are applied to the input, where the result is added to the bias and passed through a nonlinear activation function σ. For example, to compute a single neuron activation, Y = σ(b + ∑x i w i ).

[0062] In the underlying hardware, R bits of precision are used to represent each output and input activation value, represented by Y and x. i S bits of precision are used to represent each weight value, represented by w i Given. For a given network structure, each layer may require a different minimum accuracy to maintain high performance (e.g., classification accuracy). One approach is to build the hardware with the maximum accuracy required and use that accuracy for all layers. However, this approach is inefficient in terms of energy and area. Therefore, the present disclosure provides an iterative method that can achieve any necessary accuracy by trading off time for accuracy.

[0063] To increase the activation precision beyond R bits, use a modified nonlinear activation function that first computes the most significant R bits, zeroes them (e.g., by shifting), then computes the next most significant R bits, and repeats. Computing to R*M bits of precision would require M steps (possibly pipelined with subsequent operations). The limit of precision is determined by the precision of the partial sums before the nonlinearity: b+∑x i w i .

[0064] To increase the weight precision to more than S bits, an iterative approach is applied. To compute S*N bits of weight precision, N steps are used. For each step, the partial sum update is computed with S bits, starting from the most significant bit. After each update, the partial sum P is multiplied by (at most) 2 S (This can be accomplished by bit shifting) to account for the more significant bits appropriately. When updating P with the least significant S bits, it is not multiplied by 2 s .

[0065] For example, if S=1 and N=4, then S*N bits of precision weight w are stored i And use it as 4 parts:

[0066]

[0067]

[0068]

[0069]

[0070] The partial sum P is then aggregated across multiple partial sums calculations as follows:

[0071]

[0072] Therefore, the present disclosure provides a code conversion circuit that uses various scalar, vector, matrix or tensor formats and outputs a tensor format compatible with the target underlying neural network hardware. In various embodiments, high-precision data is converted to low-precision data compatible with the underlying neural network hardware. In some embodiments, input in scalar, vector, matrix or tensor format is converted to a specific vector, matrix or tensor format compatible with the underlying neural network hardware. In various embodiments, flexible precision conversion is provided. In some embodiments, flexible precision transformation is achieved through sequential iteration.

[0073] The present disclosure provides flexible precision neural network computation by using one or more low precision computational elements. In some embodiments, flexible precision is achieved by sequentially iterating the low precision elements.

[0074] Reference now Figure 5, a method for operating a neural network according to an embodiment of the present disclosure is shown. At 501, an input data tensor is received at a neural network processor including multiple neural cores. The input data tensor has a feature dimension at an input bit precision. The neural network processor is configured for one or more processor feature dimensions at one or more processor bit precisions. At 502, the input data tensor is converted from the input bit precision to one of the processor bit precisions. At 503, the input data tensor is divided into a plurality of blocks, each block conforming to one of the processor feature dimensions. At 504, each of the plurality of blocks is provided to one of the plurality of neural cores. At 505, the plurality of neural cores calculate outputs of one or more neural network layers.

[0075] Reference now Figure 6 , a schematic diagram of an example of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to imply any limitation on the scope of use or functionality of the embodiments described herein. In any case, computing node 10 can be implemented and / or perform any of the functions set forth above.

[0076] The computing node 10 has a computer system / server 12, which can operate with many other general or special computing system environments or configurations. As is well known, examples of computing systems, environments and / or configurations suitable for operating with the computer system / server 12 include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0077] Computer system / server 12 may be described in the general context of computer system executable instructions (e.g., program modules) executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0078] like Figure 6 As shown, a computer system / server 12 in a computing node 10 is shown in the form of a general-purpose computing device. Components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processor 16.

[0079] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0080] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and nonvolatile media, removable and non-removable media.

[0081] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may also include other removable / non-removable devices, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading and writing to a non-removable non-volatile magnetic medium (not shown and commonly referred to as a "hard drive"). Although not shown, a disk drive for reading and writing from a removable non-volatile disk (such as a "floppy disk"), and an optical drive for reading or writing from a removable non-volatile optical disk may provide a disk such as a CD-ROM, DVD-ROM, or other optical media. In this case, each may be connected to the bus 18 via one or more data media interfaces. As will be further depicted and described below, the memory 28 may include at least one program product having a set of program modules (e.g., at least one) configured to perform the functions of an embodiment of the present disclosure.

[0082] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data may be stored in memory 28 as an example and not by way of limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0083] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. However, the computer system / server 12 may communicate with one or more networks (such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet)) via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0084] The present disclosure may include systems, methods, and / or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform various aspects of the present disclosure.

[0085] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0087] The computer program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0088] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0091] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0092] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A calculation method, comprising: Receiving an input data tensor at a neural network processor including a plurality of neural cores, the input data tensor having a feature dimension at an input bit precision; The neural network processor is configured for one or more feature dimensions at one or more processor bit precisions; determining a number of blocks by dividing an input bit precision of the input data tensor by one of one or more processor bit precisions; determining one of the one or more processor bit precisions; converting the input data tensor from an input bit precision to the determined one of the one or more processor bit precisions based on the determined one of the one or more processor bit precisions, wherein converting the input data tensor comprises removing one or more least significant bits of the input data tensor, wherein the number of the one or more least significant bits is based on the one or more processor bit precisions; Divide the input data tensor into the number of blocks, each block conforming to one of the neural network processor feature dimensions; providing each of the plurality of blocks to one of the plurality of neural cores; and The output of one or more neural network layers is calculated by multiple neural kernels.

2. The method of claim 1, wherein the input data tensor comprises an image.

3. The method of claim 1, wherein the neural network processor is configured for a predetermined number of features, wherein the converting of the input data tensor comprises dividing the input features into a plurality of feature sets, each feature set having less than or equal to the predetermined number of features. 4 . The method of claim 1 , wherein partitioning the input data tensor comprises zero padding the number of blocks in one of the feature dimensions to conform to one of the processor feature dimensions.

5. The method of claim 1, wherein partitioning the input data tensor comprises packing the input data tensor.

6. The method of claim 5, wherein packing the input data tensor comprises: Reorganize the input features to load unused processor feature dimensions with data from non-feature dimensions of the input features.

7. The method of claim 1 , wherein the neural network processor is configured to: calculate a plurality of fixed precision partial sums; and Combines multiple fixed-precision partial sums into a full sum. The method of claim 7 , wherein the plurality of fixed precision partial sums are intermediate results.

9. The method of claim 8, wherein the intermediate result is a weighted sum of subsets of the inputs.

10. The method of claim 7, wherein the neural network processor is configured to iteratively compute the partial sums from a plurality of fixed precision partial sums.

11. A computing system comprising: A neural network processor, comprising a plurality of neural cores, wherein each activation of the neural network processor has one or more processor bit precisions, and the processor is configured to accept data having a processor characteristic dimension; A conversion circuit is coupled to the neural network processor and is adapted to: Receiving an input data tensor having input bit precision per channel on one or more features; determining one of the one or more processor bit precisions; determining a number of blocks by dividing an input bit precision of the input data tensor by one of one or more processor bit precisions; converting the input data tensor from an input bit precision to the determined one of the one or more processor bit precisions based on the determined one of the one or more processor bit precisions, wherein converting the input data tensor comprises removing one or more least significant bits of the input data tensor, wherein the number of the one or more least significant bits is based on the one or more processor bit precisions; Divide the input data tensor into the number of blocks, each block conforming to one of the neural network processor feature dimensions; providing each of the number of blocks to one of a plurality of neural cores; and The neural network processor is suitable for calculating the output of one or more neural network layers through multiple neural cores.

12. The system of claim 11, wherein the input data tensor comprises an image.

13. The system of claim 11, wherein the transforming the input data tensor comprises partitioning each channel into a plurality of values ​​having less than or equal to a processor bit precision.

14. The system of claim 11, wherein partitioning the input data tensor comprises zero padding a number of blocks of the blocks in a feature dimension to conform to a processor feature dimension.

15. The system of claim 11, wherein partitioning the input data tensor comprises packing the input data tensor.

16. The system of claim 15, wherein the packing of input data tensors comprises: Reorganize the input features to load unused processor feature dimensions with data from non-feature dimensions of the input features.

17. The system of claim 11, wherein the neural network processor is configured to: compute a plurality of fixed precision partial sums; and Combines multiple fixed-precision partial sums into a full sum.

18. The system of claim 17, wherein the plurality of fixed precision partial sums are intermediate results.

19. The system of claim 18, wherein the intermediate result is a weighted sum of subsets of the inputs.

20. The system of claim 17, wherein the neural network processor is configured to iteratively compute the partial sums from a plurality of fixed precision partial sums.

21. A computing system comprising A neural network processor comprising a plurality of neural cores, the neural network processor being configured for one or more processor feature dimensions at one or more processor bit precisions; and A conversion circuit is coupled to the neural network processor and is adapted to: receiving an input data tensor having feature dimensions at an input bit precision; determining one of one or more processor bit precisions; determining a number of blocks by dividing an input bit precision of the input data tensor by one of one or more processor bit precisions; converting the input data tensor from an input bit precision to the determined one of the one or more processor bit precisions based on the determined one of the one or more processor bit precisions, wherein converting the input data tensor comprises removing one or more least significant bits of the input data tensor, wherein the number of the one or more least significant bits is based on the one or more processor bit precisions; Divide the input data tensor into the number of blocks, each block conforming to one of the neural network processor feature dimensions; providing each of the number of blocks to one of a plurality of neural cores; and The plurality of neural cores are suitable for computing outputs of one or more neural network layers.

22. A calculation method comprising Receive input data tensor; The input data tensor has per-channel input bit precision on one or more features; Determine the processor bit accuracy of the neural network processor; determining a number of blocks by dividing an input bit precision of the input data tensor by one of one or more processor bit precisions; based on the determination of the processor bit precision of the neural network processor, converting the input data tensor from the input bit precision to the processor bit precision of the neural network processor, wherein converting the input data tensor comprises removing one or more least significant bits of the input data tensor, wherein the number of the one or more least significant bits is based on the processor bit precision; The neural network processor has multiple neural cores; The neural network processor is configured to receive data having a feature dimension; Divide the input data tensor into the number of blocks, each block conforming to one of the neural network processor feature dimensions; providing each of the number of blocks to one of the plurality of neural cores; and The neural network processor is suitable for calculating the output of one or more neural network layers through the multiple neural cores.

Images