Deriving a consistent software neural network layer from a quantized firmware neural network layer

Abstract

A system and method for deriving a consistent software neural network layer is provided. The method includes receiving a first instruction, the first instruction being configured to process a first data set corresponding to the neural network layer using a neural network processor (NNP), wherein the NNP is configured to quantize the first data set to generate a quantized data set, and then perform a matrix-vector multiplication operation on the quantized data set using a matrix-vector multiplier included in hardware associated with the NNP to generate a first result set. The method also includes processing the first instruction to automatically generate a second instruction, the second instruction being configured for use with at least one processor different from the NNP, such that the second instruction, when executed by the at least one processor to perform the matrix multiplication operation, generates a second result set consistent with the first result set.

Description

Background Art

[0001] Neural network technology is used to perform complex tasks such as image classification, reading comprehension, language translation or speech recognition. Many of these tasks involve deep learning that involves performing a large number of floating point matrix multiplication and accumulation operations. These operations are performed during training and during the provision of results based on input data and trained data.

[0002] Neural networks can use values ​​corresponding to input data and training data expressed in different formats, including data expressed at different levels of precision. Summary of the invention

[0003] In one example, the present disclosure relates to a method, the method comprising receiving a first instruction, the first instruction being configured to process a first data set corresponding to a neural network layer using a neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a quantized data set, and then using a matrix-vector multiplier included in hardware associated with the neural network processor, perform a matrix-vector multiplication operation on the quantized data set to generate a first result set. The method also includes processing the first instruction to automatically generate a second instruction, the second instruction being configured for use with at least one processor different from the neural network processor, such that the second instruction, when executed by the at least one processor to perform the matrix multiplication operation corresponding to the neural network layer, generates a second result set consistent with the first result set.

[0004] In another example, the present disclosure relates to a system including at least one processor and a memory. The memory may include first instructions configured to process a first data set corresponding to a neural network layer using a neural network processor, the neural network processor having a matrix-vector multiplier included in hardware associated with the neural network processor and a multifunction unit included in hardware associated with the neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a first quantized data set, and then: (1) perform a matrix operation on the first quantized data set using the matrix-vector multiplier included in hardware associated with the neural network processor to generate a first output data set, (2) quantize the first output data set to generate a first quantized output data set, and (3) perform a scalar operation on the first quantized output data set using the multifunction unit in hardware associated with the neural network processor to generate a second output data set. The memory may also include second instructions configured to process the first instructions to generate third instructions configured for use with at least one processor other than the neural network processor, wherein the third instructions include instructions for performing a matrix multiplication operation and instructions for performing a scalar operation to process the neural network layer.

[0005] In yet another example, the present disclosure relates to a non-transitory computer-readable medium including code corresponding to a method. The method may include receiving a first instruction, the first instruction being configured to process a first data set corresponding to a neural network layer using a neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a quantized data set, and then use a matrix-vector multiplier included in hardware associated with the neural network processor to perform a matrix-vector multiplication operation on the quantized data set to generate a first result set. The method also includes processing the first instruction to automatically generate a second instruction, the second instruction being configured for use with at least one processor different from the neural network processor, such that the second instruction, when executed by the at least one processor to perform a matrix multiplication operation corresponding to the neural network layer, generates a second result set consistent with the first result set.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure is illustrated by way of example and not limitation in the accompanying figures, in which like reference numerals represent like elements. The elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.

[0008] Figure 1 is a block diagram of an environment for automatically generating consistent trained software models according to an example;

[0009] Figure 2 A block diagram of a portion of the quantized firmware neural network layer;

[0010] Figure 3 is a block diagram of a neural network processor with quantization according to an example;

[0011] Figure 4 A block diagram of a portion of the quantized software neural network layer;

[0012] Figure 5 is a block diagram of a system for automatically generating instructions corresponding to a consistent software neural network layer according to one example;

[0013] Figure 6 A flow chart of a method for automatically generating instructions corresponding to a consistent software neural network layer according to one example is shown. DETAILED DESCRIPTION

[0014] Examples disclosed in the present disclosure relate to systems, methods, and components for implementing neural network-based processing. Certain examples relate to processing convolutional neural network (CNN) layers using quantization. Certain examples relate to processing CNN layers using a neural network processor. Certain examples relate to maintaining consistency between quantized firmware neural network layers and software neural network layers. The neural network processor can be implemented using any of the following devices: a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an erasable and / or complex programmable logic device (PLD), a programmable array logic (PAL) device, and a general array logic (GAL) device. The neural network processor can also be implemented using a CPU, a GPU, a combination of a CPU and a GPU, or any combination of programmable hardware, a CPU, and a GPU. An image file can be used to configure or reconfigure an FPGA. An image file or a similar file or program can be delivered from a host CPU via a network link or a local link (e.g., PCIe). The information included in the image file can be used to program the hardware blocks of a node (e.g., logic blocks and reconfigurable interconnects of an FPGA) to implement the intended functionality. The intended functionality may be implemented to support any service that may be provided via a combination of computing, network, and storage resources, such as via a data center or other infrastructure for delivering services.

[0015] The described multiple aspects can also be implemented in a cloud computing environment. Cloud computing can refer to a model for initiating on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be deployed in a market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be quickly pre-allocated via virtualization and released with a lower management workload or service provider interaction, and then scaled accordingly. The cloud computing model can be composed of various features, such as on-demand self-service, extensive network access, resource pools, rapid elasticity, measurable services, etc. The cloud computing model can be used to expose various service models, such as, for example, hardware as a service ("HaaS"), software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0016] Machine learning services, such as those based on recurrent neural networks (RNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), can be implemented using the systems and nodes described in the present disclosure. In one example, content or other information related to the service, such as words, sentences, images, videos, or other such content / information, can be converted into vector representations.

[0017] In one example, a neural network model may include many layers and each layer may be encoded as a matrix or vector of weights expressed in the form of coefficients or constants acquired through training of the neural network. A GPU or programmable hardware logic block in a node may process the matrix or vector to perform various operations, including multiplication, addition, and other operations on an input vector representing encoded information related to a service.

[0018] Taking LSTM as an example, an LSTM network can contain a series of repeated RNN layers or other types of layers. Each layer of the LSTM network can consume input at a given time step, for example, the state of the layer from the previous time step, and can produce a new set of outputs or states. In the case of LSTM, a single chunk of content can be encoded as a single vector or multiple vectors. As an example, a word or a combination of words (e.g., a phrase, sentence, or paragraph) can be encoded as a single vector. Each chunk can be encoded into a separate layer (e.g., a specific time step) of the LSTM network. The LSTM layer can be described using a system of equations, such as the following:

[0019] i t =σ(W xi xt+W hi h t-1 +W ci c t-1 +b i

[0020] f t =σ(W xf x t +W hf h t-1 +W cf c t-1 +b f )

[0021] c t =f t c t-1 i t tanh(W xc x t +W hc h t-1 +b c )

[0022] o t =σ(W xo x t +W ho h t-1 +W co c t +b o )

[0023] h t =o t tanh(c t )

[0024] In this example, within each LSTM layer, the input and hidden states can be processed using a combination of vector operations (e.g., dot products, inner products, or vector additions) and nonlinear functions (e.g., sigmoids, hyperbolic, and tangent). In some cases, the most computationally intensive operations can occur from dot products, which can be implemented using dense matrix-vector and matrix-matrix multiplication routines. In one example, the processing of vector operations and nonlinear functions can be performed in parallel.

[0025] Values ​​corresponding to training data (including vector data) can be represented in a digital format. Floating point representations for vector data values ​​are expensive because each individual point value has an exponent specific to that point value. An alternative may be a fixed point representation. Performance, energy usage, and storage requirements can be improved by using a reduced precision format to implement an artificial neural network. This format can represent floating point numbers using a small (e.g., 3, 4, or 5 bits) mantissa and an exponent shared by two or more floating point numbers. A neural network using a reduced precision format may be referred to as a quantized neural network.

[0026] In one example, a fixed point representation can express a number using a set of integer bits and fractional bits. Fixed point can be efficiently processed in hardware using integer arithmetic, which can make it a preferred format when applicable. The fixed point format can be represented as qX.Y, where X is the integer bit number and Y is the fractional bit number. Block floating point (BFP) can apply a shared exponent to a block of fixed point numbers; for example, a vector or matrix. The shared exponent can allow a significantly higher dynamic range for the block, even though the individual block members have fixed ranges relative to each other.

[0027] Quantized neural networks can improve the latency and throughput of running neural networks by reducing computational and memory requirements. However, the use of reduced precision formats (e.g., any of reduced precision floating point formats, block floating point (BFP), or integers) can cause problems when training neural networks. As the neural network converges, parameters that may be appropriate at the beginning of training may become suboptimal. In addition, many neural network methods typically use full-precision floating point numbers (e.g., 32-bit or 16-bit floating point numbers). As an example, some software implementations of neural networks can use full-precision floating point numbers. On the other hand, some hardware implementations of neural networks can use reduced precision numbers. Because the underlying implementations of software and hardware accelerated neural networks are different, small differences in calculations can occur, which can cause errors over time.

[0028] The core of many neural network algorithms is the matrix multiplication operation: y=xW, where x is the input activation and W is the weight matrix. In one example, quantization is applied to the input activation and weight matrix to reduce the hardware cost of computing the matrix multiplication. Therefore, the matrix multiplication operation can be y=Q(x)Q(W). Using unquantized matrix multiplication Arbitrary scalar values ​​can be factored without affecting the result. However, if quantization is applied to this form, the quantized matrix multiplication The result is affected by two aspects. First, the quantized weights have different values, so they can lead to different quantized point sets. Second, the high-precision scalar multiplication after the matrix multiplication can also change the result.

[0029] Quantized data - e.g., data in a block floating point format (e.g., with a reduced size mantissa and / or shared exponent) - can be used to represent weights or activation values ​​in both software and hardware / firmware (e.g., an FPGA implementation) implementations of a neural network model. Because the underlying algorithms between the implementations are different, the simulation results differ from the actual results in the hardware / firmware implementation. Certain disclosed examples relate to generating consistent results for a neural network that is implemented using software using quantized data and implemented using hardware / firmware that also uses quantized data. One example of the disclosed technology provides a software implementation that matches the actual operations performed on the hardware / firmware. Thus, the software and hardware / firmware implementations can advantageously have the same results. In some examples, a software implementation that matches the actual operations performed in the hardware / firmware is referred to as a consistent software implementation of a neural network model.

[0030] Figure 11 is a block diagram of an environment 100 for automatically generating a consistent software model 140 according to an example. In this example, the trained software model 110 may include instructions and other software used to express a neural network layer. As an example, the trained software model 110 may include instructions including calculations (e.g., matrix multiplication, ReLU, softmax). These calculations are expressed in a stateful data flow graph using software (such as TensorFlow). The full-precision trained firmware model 120 may refer to instructions and hardware / firmware implementations of the neural network model. In one example, the weights and activation values ​​corresponding to the trained software model 110 and the full-precision trained firmware model 120 may be expressed in a format such as a floating point format. In contrast, the quantized firmware model 130 may refer to instructions and hardware / firmware implementations of a neural network layer using a neural network processor using quantized data, and the quantized data may be expressed in a reduced precision format. In one example, the neural network processor may be configured to quantize data to generate quantized data, and then perform matrix operations on the quantized data using a matrix-vector multiplier contained within the hardware associated with the neural network processor. For additional details on an implementation of a neural network processor using quantized data, see Figure 3 In one example, the consistent software model 140 may include instructions configured to be processed by a compiler for execution using at least one processor other than the neural network processor, such that despite using quantized data, the results produced by the instructions are consistent with a neural network model implemented using hardware / firmware.

[0031] Figure 2 2 is a block diagram of a portion of a quantized firmware neural network layer 200. The portion includes a neural network portion (NNP) 210 and another neural network portion (NNP) 250. NNP 210 illustrates operations that may be performed on weight values ​​A0 and A1, input vector values ​​x0 and x1, and bias vector value b0. In this example, weight values ​​A0 and A1 may be stored in an N×N matrix, where N is the number of neural network processors (e.g., NNPs) used with the quantized firmware neural network layer 200. Figure 3 214). The input values ​​x0 and x1 may be stored in an N×1 matrix. NNP 210 illustrates both a multiplication and accumulation (MAC_0 212 and MAC_1 214) operation and an addition operation (Add_0 216). Weight values ​​A0 and A1 may be quantized (Q 222 and Q 232, respectively) prior to the multiplication and accumulation operation. Input values ​​x0 and x1 may also be quantized (Q 224 and Q 234, respectively) prior to the multiplication and accumulation operation. The output generated by the multiplication and accumulation (MAC_0 212) operation may be quantized (Q 226).

[0032] Continue to refer Figure 2 , NNP 250 shows operations that can be performed on weight values ​​A2 and A3, input vector values ​​x0 and x1, and bias vector value b1. In this example, weight values ​​A2 and A3 can also be stored in an N×N matrix, where N is the native dimension of the neural network processor used with the quantization firmware neural network layer 200. Input values ​​x0 and x1 can also be stored in an N×1 matrix. NNP 250 shows multiplication and accumulation (MAC_0 252 and MAC_1 254) operations and addition operations (Add_0 256). Weight values ​​A2 and A3 can be quantized (Q 262 and Q 272, respectively) before the multiplication and accumulation operations. The output generated by the multiplication and accumulation (MAC_0 252) operation can be quantized (Q266). Quantization can involve mapping continuous or high-precision values ​​to a discrete, low-precision grid. If the original points are close to the quantized values ​​to which they are mapped, then one expects that the resulting calculations will be close to the original calculations. Logarithmic quantization can be used to perform quantization. As mentioned previously, converting values ​​from a higher precision format to a lower precision format in the various ways mentioned previously results in quantization. Figure 2 A certain number of neural network parts are shown, but the quantized firmware neural network layer 200 may include additional parts. Figure 2 Certain values ​​are shown to be quantized, but additional or fewer values ​​may be quantized. As an example, although Figure 2 The outputs of the multiply and accumulate (MAC_1 214 and MAC_3 254) are shown not being quantized, but they may be quantized.

[0033] Figure 3300 is a block diagram of a neural network processor 300 with quantization according to one example. Each neural network processor 300 may include an input message processor (IMP) 304 for receiving messages from other processors and an output message processor (OMP) 306 for processing outgoing messages to other processors or components. Such messages may be received and transmitted via a network 302. Each neural network processor 300 may also include a matrix vector multiplier (MVM) 310 and two or more multi-function units (MFUs) (e.g., MFU[0] 340 and MFU[1] 360). Each neural network processor 300 may also include a matrix memory manager 370, a vector memory manager 380, a vector DRAM 382, ​​and a matrix DRAM 384. In this example, the processor may accept off-chip messages containing auxiliary information, such as control and scalar data, as well as payload data (e.g., vectors, matrices, or other tensor data structures). In this example, incoming messages may be handled by a lightweight input message processor (IMP) 304, which sends vectors to a vector memory manager 380. IMP 304 may send the matrix to matrix memory manager 370 .

[0034] Each of the vector data (e.g., data corresponding to activations) and the matrix data (e.g., data corresponding to weights) may be quantized using vector quantization 392 and matrix quantization 394, respectively. Thus, the vector data received from the vector memory manager 380 may be in a higher precision format (e.g., FP16 or FP32) and the vector quantization 392 may convert the vector data from the higher precision format to a lower precision format (e.g., a block floating point format). Similarly, the matrix data received via the network 302 or otherwise may be in a higher precision format (e.g., FP16 or FP32) and the matrix quantization 394 may convert the matrix data from the higher precision format to a lower precision format (e.g., a block floating point format). Since matrix multiplication operations are more expensive in terms of resources and time, in one example, it may be advantageous to quantize only the inputs to the MVM 310. Thus, in Figure 3 In the example shown, only the input to the MVM 310 is quantized. The input to the MFU is not scaled and quantized. Alternatively, in another example, the input to both the MVM 310 and the MFU may be quantized.

[0035] Continue to refer Figure 3, each matrix in the matrix can have a size of N×N and each vector in the vector can have a size of 1×N. In this example, all instructions corresponding to the neural network processor 300 can operate on data of native size. The logical vectors and matrices corresponding to the applications handled by the neural network processor 300 can generally be larger than the native size; in these cases, the vectors and matrices can be decomposed into tiles of native size. In one example, the block size of the BFP format data can be equal to the native dimension. Therefore, each native 1×N vector can have a shared exponent, and each row of the N×N matrix can have a shared exponent. Each of the vector data and the matrix data can have a mantissa portion of a two's complement code, and the size of the mantissa for the vector data and the matrix data can be different. In one example, a shared exponent may not be required. Instead, the weight values ​​can be fixed within an integer range, for example, -2 b -1 to 2 b -1. This can further simplify the hardware used to perform the dot product operation.

[0036] Still reference Figure 3 , MVM 310 may include a vector register file (VRF) 312, a matrix register file (MRF) 320, and a tile engine (e.g., tile engines 314, 316, and 318). The tile engine may receive input matrix and input vector data from VRF 312. MVM 310 may also include an accurate format converter. In this example, a quantization operation may be performed for each tile. Thus, matrix quantization 394 may quantize weight values ​​for each of tile engines 314, 316, and 318 in parallel. Similarly, vector quantization 392 may quantize activations or input values ​​for each of tile engines 314, 316, and 318 in parallel. In one example, two internal BFP formats may be used by MVM 310 to express its input and output: BFP short, for vector and matrix storage, and BFP long, for accumulation. In one example of MVM 310, BFP short may use q1.15 fixed point value and shared 5-bit exponent, while BFP long may use q34.40 fixed point value and shared 5-bit exponent. In this example, matrix-vector multiplication may result in BFP long, which may be converted back to floating point format as the final output stage. Therefore, Figure 3 The example MVM 310 shown in FIG. 3 may include BFP to FP16 converters 322, 324, and 326 at the output stage. The tile engines 314, 316, and 318 may provide outputs to respective converters in parallel, such as Figure 3 as shown in the example.

[0037] Matrix data may be transferred between the matrix DRAM 384 and the matrix memory manager 370 using M channels. The vector memory manager 380 may move vector data over C channels.

[0038] Continue to refer Figure 3 , each MFU (e.g., MFU[0] 340 and MFU[1] 360) may include a crossbar (e.g., a crossbar labeled xbars). MFU[0] 340 may support vector operations such as vector-vector multiplication and addition, Sigmoid function, TanH function, softmax operation, rectified linear unit (ReLU) operation, and / or activation block operation. Thus, Figure 3 As shown, MFU[0]340 may include a crossbar (e.g., xbar 346, 348, and 350) that streams vectors from its input bus through a pipeline sequence of operations. Thus, vectors may be received via a register file labeled MulVrf 342 or another register file labeled AsVrf[0]344, and such vectors may be affected by any of a multiplication operation, an addition operation, or some other operation. MFU[0]340 may include several hardware blocks (e.g., 353, 357, and 361) for performing addition. MFU[0]340 may also include several hardware blocks (e.g., 352, 356, and 359) for performing multiplication. MFU[0]340 may also include several hardware blocks (e.g., 351, 354, and 358) for performing activation.

[0039] Still reference Figure 3 , MFU[1] 360 may include a multiplexing circuit 374 that may allow MFU[1] 360 to receive outputs from MFU[0] 340 and perform additional operations on those outputs and any additional inputs received via ADD / SUB VRF 368. MFU[1] 360 may include several hardware blocks (e.g., 369, 371, and 372) for performing additions. MFU[1] 360 may also include several hardware blocks for performing activations. Outputs from MFU[1] 360 received via the C channel may be coupled to a vector memory manager 380 via multiplexing circuit 374. Although Figure 3 A certain number of components of the neural network processor 300 are shown arranged in a certain manner, but there may be a greater or lesser number of components arranged differently.

[0040] The neural network processor 300 can be used to enable the issuance of instructions that trigger millions of operations using a relatively small number of instructions. As an example, the following Table 1 shows instructions corresponding to a fully parameterized LSTM:

[0041]

[0042]

[0043] Table 1

[0044] Although Table 1 shows a certain number of instructions having certain formats, the neural network processor 300 may execute more or fewer instructions having different formats to achieve the same goal.

[0045] Table 2 below shows how a 1×1 convolution is calculated as part of a CNN evaluation.

[0046]

[0047] Table 2

[0048] As shown in the table above, the number of iterations over the instruction chain for calculation can be specified. Next, the native dimension of each instruction chain can be scaled by the column scaling factor as needed. And after reading the vector data from the vector register file, it can be multiplied with the weights retrieved from the matrix register file. After performing the additional operations required for CNN evaluation, the output can be provided. As an example, a point-by-point rectified linear unit (ReLU) operation can be performed for each element of the vector data.

[0049] Table 3 below shows how to calculate N×N convolutions as part of CNN evaluation. The following instructions similar to 1×1 convolutions are not repeated. The Set2dWindows instruction can be used to set the total window size, and then the SetIterations instruction can be used to slide the window across the input volume. The *_inc instructions (e.g., v_rd_inc and v_add_inc) can be used to increase the address of the instruction based on the stride. As an example, a stride of 2 can cause a skip of every other vector in a vector register file used to store vector data for operations such as addition.

[0050]

[0051] Table 3

[0052] although Figure 3 A neural network processor is shown with specific components, including a matrix-vector multiplier and a multifunction unit, but the methods and concepts described herein may be included as part of other processors configured to perform operations related to neural network training. As an example, any processor configured to perform operations such as multiply and accumulate operations, addition operations, or other such operations may perform quantization and thus may require systems and related methods for generating consistent neural network instructions or operations.

[0053] Figure 4 4 is a block diagram of a portion of a quantized software neural network layer 400. The portion includes a neural network portion (NNP) 410 and another neural network portion (NNP) 450. NNP 410 shows operations that can be performed on weight values ​​A0 and A1, input vector values ​​x0 and x1, and bias vector value b0. NNP 410 shows both matrix multiplication (MatMul 412 and MatMul414) operations and addition operations (Add 416). Weight values ​​A0 and A1 can be quantized (Q 422 and Q 432, respectively) before the matrix multiplication operation. Input values ​​x0 and x1 can also be quantized (Q 424 and Q 434, respectively) before the matrix multiplication operation. The output generated by the matrix multiplication (MatMul 412) operation can be quantized (Q 426).

[0054] Continue to refer Figure 4 , NNP 450 shows the operations that can be performed on weight values ​​A2 and A3, input vector values ​​x0 and x1, and bias vector value b1. NNP 450 shows both matrix multiplication (MatMul 452 and MatMul 454) operations and addition operations (Add 456). Weight values ​​A2 and A3 can be quantized (Q 462 and Q472, respectively) before the matrix multiplication operation. Input values ​​x0 and x1 can also be quantized (Q 464 and Q 474, respectively) before the matrix multiplication operation. The output generated by the matrix multiplication (MatMul 452) operation can be quantized (Q 466). Quantization can involve mapping continuous or high-precision values ​​to discrete, low-precision grids. If the original points are close to the quantized values ​​to which they are mapped, then one would expect the resulting calculations to be close to the original calculations. Logarithmic quantization can be used to perform quantization. As previously described, the conversion of values ​​from a higher-precision format to a lower-precision format in the various ways mentioned above can cause quantization. Although Figure 4 A certain number of neural network parts are shown, but a portion of the quantized software neural network layer 400 may include additional parts. Figure 4 Although certain values ​​are shown to be quantized, additional or fewer values ​​may be quantized. As an example, although Figure 4 The outputs of the matrix multiplication (MatMul 414 and MatMul 454) operations are shown not being quantized, but they may be quantized.

[0055] Figure 55 is a block diagram of a system 500 for automatically generating instructions corresponding to a consistent software neural network layer according to one example. The system 500 may include a processor 510, a memory 520, an input / output device 540, a display 550, and a network interface 560 interconnected via a bus system 502. The memory 520 may include firmware neural network processing (NNP) code 522, software neural network processing (NNP) code 524, and consistency conversion code 526. Table 4 below shows instructions that may be part of the firmware NNP code 522, which is configured to process such as Figure 2 5. In this example, the instructions include additional information in addition to the instructions that can be executed by the neural network processor 300. As an example, the firmware NNP code 522 can include hints.

[0056]

[0057]

[0058] Table 4

[0059] Table 5 below includes example consistency conversion code 526. Consistency conversion code 526 may include: (1) instructions for parsing firmware NNP code 522, and (2) instruction handlers for implementing operations associated with a firmware neural network processor.

[0060]

[0061]

[0062]

[0063]

[0064] Table 5

[0065] In addition, data manipulation functions for handling differences in how data is processed between the neural network processor 300 and another processor may be needed. As an example, data manipulation functions can be used to convert matrices with native dimensions of the neural network processor 300 into data suitable for consumption by software instructions corresponding to another processor. In one example, this can include zero padding the data so that the data organized in the native dimension format becomes a format suitable for software instructions (e.g., TensorFlow instructions). Table 6 below shows example code for data set preparation.

[0066]

[0067]

[0068]

[0069] Table 6

[0070] Table 7 below shows an example of software neural network processing (NNP) code 524.

[0071] This example shows the code for TensorFlow.

[0072]

[0073]

[0074]

[0075]

[0076] Table 7

[0077] although Figure 5 Some number of components of the system 500 arranged in a certain manner are shown, but additional or fewer components of different arrangements may also be used. In addition, although the memory 520 shows some code blocks, the functions provided by the code can be combined or distributed. In addition, various code blocks can be stored in non-transient computer-readable media, such as non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, disks or tapes, optical disks or tapes, flash memory, EPROM, NVRAM, PRAM or other such media, or network versions of such media. Volatile media include, for example, dynamic memory, such as DRAM, SRAM, cache or other such media.

[0078] Furthermore, as used herein, the term instruction is not limited to "instructions" expressed in a particular encoding or via a particular syntax. As an example, an instruction may include a state machine or other form of encoding that may be used to capture the operations that the instruction may perform. Thus, in summary, regardless of the expression or nature of the instruction, if a neural network layer is implemented in hardware and involves quantized operations, then it may be used. Figure 5 system 500.

[0079] Figure 6A flowchart 600 of a method for automatically generating instructions corresponding to a consistent software neural network model according to an example is shown. Step 610 may include receiving a first instruction, the first instruction being configured to process a first data set corresponding to a neural network layer using a neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a quantized data set, and then using a matrix-vector multiplier included in hardware associated with the neural network processor to perform a matrix-vector multiplication operation on the quantized data set to generate a first result set. The first instruction may be associated with Figure 5 The instructions may correspond to the firmware NNP code 522. These instructions may be received via any input / output device 540 or via the network interface 560. As previously described, these instructions may be configured for use with the neural network processor 300.

[0080] Step 620 may include processing the first instruction to automatically generate a second instruction, the second instruction being configured for use with at least one processor other than the neural network processor, such that the second instruction, when executed by the at least one processor to perform a matrix multiplication operation corresponding to a neural network layer, generates a second result set that is consistent with the first result set. As part of this step, in one example, consistency conversion code 526 may be executed by processor 510 to process firmware NNP code 522. This processing may cause automatic generation of the second instruction, which may be Figure 5 524 of software NNP code. Step 620 may include extracting information about the dependency between the matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation. Thus, in this example, the consistency transformation code 526 may extract information about the dependency between the operations. This information in turn may be used to generate a second instruction (e.g., software NNP code 524). As an example, the extracted dependency information may be used to create a data flow dependency as part of the software NNP code 524.

[0081] although Figure 6 Several steps are described as being performed in a particular order, but more or fewer steps may be performed in a different order.

[0082] In summary, the present disclosure relates to a method, including receiving a first instruction, the first instruction being configured to process a first data set corresponding to a neural network layer using a neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a quantized data set, and then using a matrix-vector multiplier included in hardware associated with the neural network processor, to perform a matrix-vector multiplication operation on the quantized data set to generate a first result set. The method also includes processing the first instruction to automatically generate a second instruction, the second instruction being configured for use with at least one processor different from the neural network processor, such that the second instruction, when executed by the at least one processor to perform the matrix multiplication operation corresponding to the neural network layer, generates a second result set consistent with the first result set.

[0083] Processing the first instruction to automatically generate the second instruction also includes: extracting information about a dependency between a matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation. Processing the first instruction to automatically generate the second instruction may include converting the form of the first data set into another form suitable for use with the second instruction. The quantized data set may include a quantized training data set for use with an operation associated with the second instruction.

[0084] The first data set is represented in a first precision format having a first precision, and the quantized data set is represented in a second precision format having a second precision lower than the first precision. The first precision format comprises a floating point format, and wherein the second precision format comprises a precision format selected from the following precision formats: an integer format, a reduced floating point precision format, or a block floating point format.

[0085] The first data set is organized in an N×N matrix form, and wherein N is an integer greater than 1 and N is a native dimension associated with the matrix-vector multiplier, and wherein processing the first instruction to automatically generate the second instruction may include: converting the first data set from the N×N matrix form to another form suitable for use with the second instruction.

[0086] In another example, the present disclosure relates to a system including at least one processor and a memory. The memory may include first instructions configured to process a first data set corresponding to a neural network layer using a neural network processor, the neural network processor having a matrix-vector multiplier included in hardware associated with the neural network processor and a multifunction unit included in hardware associated with the neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a first quantized data set, and then: (1) perform a matrix operation on the first quantized data set using the matrix-vector multiplier included in hardware associated with the neural network processor to generate a first output data set, (2) quantize the first output data set to generate a first quantized output data set, and (3) perform a scalar operation on the first quantized output data set using the multifunction unit in hardware associated with the neural network processor to generate a second output data set; the memory may also include second instructions configured to process the first instructions to generate third instructions configured for use with at least one processor other than the neural network processor, wherein the third instructions include instructions for performing a matrix multiplication operation and instructions for performing a scalar operation to process the neural network layer.

[0087] The second instruction is also configured to extract information about a dependency between a matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation. The second instruction also includes an instruction configured to transform a form of the first data set into another form suitable for use with the third instruction. The first data set is represented in a first precision format having a first precision, and wherein each of the first quantized data set and the first quantized output data set is represented in a second precision format having a second precision lower than the first precision. The first precision format includes a floating point format, and wherein the second precision format includes a precision format selected from the following precision formats: an integer format, a reduced floating point precision format, or a block floating point format.

[0088] The quantized data set includes a quantized training data set for use with an operation associated with the second instruction. The first data set is organized in an N×N matrix form, and wherein N is an integer greater than 1 and N is a native dimension associated with a matrix-vector multiplier, and wherein the second instruction also includes an instruction configured to transform the first data set from the N×N matrix form into another form suitable for use with the third instruction.

[0089] In yet another example, the present disclosure relates to a non-transitory computer-readable medium including code corresponding to a method. The method may include receiving a first instruction, the first instruction being configured to process a first data set corresponding to a neural network layer using a neural network processor, wherein the neural network processor is configured to quantize the first data set to generate a quantized data set, and then using a matrix-vector multiplier included in hardware associated with the neural network processor to perform a matrix-vector multiplication operation on the quantized data set to generate a first result set. The method also includes processing the first instruction to automatically generate a second instruction, the second instruction being configured for use with at least one processor different from the neural network processor, such that the second instruction, when executed by the at least one processor to perform the matrix multiplication operation corresponding to the neural network layer, generates a second result set consistent with the first result set.

[0090] Processing the first instruction to automatically generate the second instruction also includes: extracting information about a dependency between a matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation. The first data set is represented in a first precision format having a first precision, and the quantized data set is represented in a second precision format having a second precision lower than the first precision. The first precision format includes a floating point format, and wherein the second precision format includes a precision format selected from the following precision formats: an integer format, a reduced floating point precision format, or a block floating point format.

[0091] The quantized data set may include a quantized training data set for use with an operation associated with the second instruction. The first data set is organized in an N×N matrix form, and wherein N is an integer greater than 1 and N is a native dimension associated with a matrix-vector multiplier, and wherein processing the first instruction to automatically generate the second instruction may include converting the first data set from the N×N matrix form to another form suitable for use with the second instruction.

[0092] It should be understood that the methods, modules and components described herein are merely exemplary. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc. In an abstract but still clear sense, any arrangement of components that implement the same function is effectively "associated" to achieve the desired function. Therefore, any two components combined herein to achieve a specific function can be considered to be "associated" with each other to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two components so associated may also be considered to be "operably connected" or "coupled" to each other to achieve the desired function.

[0093] Functionality associated with some of the examples described in this disclosure may also include instructions stored in non-transient media. The term "non-transient media" as used herein refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transient media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, disks or tapes, optical disks or tapes, flash memory, EPROM, NVRAM, PRAM or other such media, or network versions of such media. Volatile media include, for example, dynamic memories such as DRAM, SRAM, cache or other such media. Non-transient media are different from transmission media, but can be used in conjunction with transmission media. Transmission media are used to transmit data and / or instructions to or from a machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media, such as radio waves.

[0094] In addition, those skilled in the art will recognize that the boundaries between the functions of the above-mentioned operations are merely illustrative. The functions of multiple operations can be combined into a single operation, and / or the functions of a single operation can be distributed in additional operations. In addition, alternative embodiments can include multiple instances of a particular operation, and the order of the operations can be changed in various other embodiments.

[0095] Although the present disclosure provides specific examples, various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure. Any benefits, advantages, or solutions to problems described herein with respect to specific examples should not be construed as a key, required, or essential feature or element of any or all the claims.

[0096] Furthermore, the terms "a" or "an" as used herein are defined as one or more than one. Furthermore, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be interpreted as implying that the introduction of another claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to an invention containing only one such element, even if the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article, such as "a" or "an". The same applies to the use of definite articles.

[0097] Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements such terms describe. Therefore, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.

Claims

1. A method for deriving a consistent software neural network layer, comprising: Receiving firmware code corresponding to a neural network layer, wherein a neural network processor is configured to quantize a first data set to generate a quantized data set, and then perform a matrix-vector multiplication operation on the quantized data set using a matrix-vector multiplier included in hardware associated with the neural network processor to generate a first result set; and The firmware code corresponding to the neural network layer is converted into a consistent software code using a consistent conversion code, wherein the consistent software code is configured for use with at least one processor different from the neural network processor, so that when the consistent software code is executed by the at least one processor to perform the matrix multiplication operation corresponding to the neural network layer, it generates a second result set that is consistent with the first result set.

2. The method of claim 1, wherein converting the firmware code corresponding to the neural network layer into the conforming software code further comprises: Information about a dependency between the matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation is extracted. 3 . The method of claim 1 , wherein the first data set is represented in a first precision format having a first precision, and the quantized data set is represented in a second precision format having a second precision lower than the first precision.

4. The method of claim 3, wherein the first precision format comprises a floating point format, and wherein the second precision format comprises a precision format selected from one of: an integer format, a reduced floating point precision format, or a block floating point format.

5. The method of claim 1, wherein the quantized data set comprises a quantized training data set for use with operations associated with the conforming software code.

6. The method of claim 1 , wherein the first data set is organized in an N×N matrix form, and wherein N is an integer greater than 1 and N is a native dimension associated with the matrix-vector multiplier, and wherein converting firmware code corresponding to a neural network layer to conforming software code comprises: The first data set is converted from the N x N matrix form into another form suitable for use with the conforming software code.

7. The method of claim 1 , wherein converting firmware code corresponding to the neural network layer into conformant software code comprises: The first data set is transformed from a form into another form suitable for use with the conforming software code.

8. A system for deriving a consistent software neural network layer, comprising: at least one processor; as well as Memory, including: Firmware code corresponding to a neural network layer, the firmware code configured to quantize a first data set using a neural network processor to generate a first quantized data set, the neural network processor having a matrix-vector multiplier included in hardware associated with the neural network processor and a multifunction unit included in the hardware associated with the neural network processor, and then: (1) perform matrix operations on the first quantized data set using the matrix-vector multiplier included in the hardware associated with the neural network processor to generate a first output data set, (2) quantize the first output data set to generate a first quantized output data set, and (3) perform scalar operations on the first quantized output data set using the multifunction unit included in the hardware associated with the neural network processor, to generate a second output data set; as well as and conforming software code configured to process the firmware code to generate conforming software code configured for use with the at least one processor different from the neural network processor, wherein the conforming software code includes instructions for performing matrix multiplication operations and instructions for performing scalar operations to process the neural network layers.

9. A system according to claim 8, wherein the consistency software code is further configured to: extract information about the dependency between the matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation.

10. The system of claim 8, wherein the first data set is represented in a first precision format having a first precision, and wherein each of the first quantized data set and the first quantized output data set is represented in a second precision format having a second precision lower than the first precision.

11. The system of claim 10, wherein the first precision format comprises a floating point format, and wherein the second precision format comprises a precision format selected from one of: an integer format, a reduced floating point precision format, or a block floating point format.

12. The system of claim 8, wherein the quantized data set comprises a quantized training data set for use with operations associated with the conforming software code.

13. A system according to claim 8, wherein the first data set is organized in an N×N matrix form, and wherein N is an integer greater than 1 and N is a native dimension associated with the matrix-vector multiplier, and wherein the conforming software code also includes instructions configured to convert the first data set from the N×N matrix form to another form suitable for use with the conforming software code.

14. The system of claim 8, wherein the conforming software code further comprises instructions configured to transform a form of the first data set into another form suitable for use with the conforming software code.

15. A non-transitory computer-readable medium comprising code corresponding to a method, the method comprising: receiving firmware code corresponding to a neural network layer, wherein a neural network processor is configured to quantize a first data set to generate a quantized data set, and then perform a matrix-vector multiplication operation on the quantized data set using a matrix-vector multiplier included in hardware associated with the neural network processor, to generate a first result set; and Using a consistency conversion code, the firmware code corresponding to the neural network layer is converted into a consistency software code, and the consistency software code is configured for use with at least one processor different from the neural network processor, so that when the consistency software code is executed by the at least one processor to perform the matrix multiplication operation corresponding to the neural network layer, it generates a second result set that is consistent with the first result set.

16. The non-transitory computer readable medium of claim 15, wherein converting the firmware code corresponding to the neural network layer to the conforming software code further comprises: Information about a dependency between the matrix-vector multiplication operation and an operation selected from a softmax operation, a ReLU operation, or an addition operation is extracted. 17 . The non-transitory computer readable medium of claim 15 , wherein the first data set is represented in a first precision format having a first precision, and the quantized data set is represented in a second precision format having a second precision lower than the first precision.

18. The non-transitory computer-readable medium of claim 17, wherein the first precision format comprises a floating point format, and wherein the second precision format comprises a precision format selected from one of: an integer format, a reduced floating point precision format, or a block floating point format.

19. The non-transitory computer-readable medium of claim 15, wherein the quantized data set comprises a quantized training data set for use with operations associated with the conforming software code.

20. The non-transitory computer readable medium of claim 15, wherein the first data set is organized in an N×N matrix form, and wherein N is an integer greater than 1 and N is a native dimension associated with the matrix-vector multiplier, and wherein converting firmware code corresponding to a neural network layer to conforming software code comprises: The first data set is converted from the N x N matrix form into another form suitable for use with the conforming software code.

Images