Sparsification of narrow data formats for neural networks

Abstract

Embodiments of the present disclosure include systems and methods for sparsifying a narrow data format for a neural network. A plurality of activation values ​​in the neural network are provided to a multiplexing unit. A set of sparsifying operations are performed on the plurality of weight values ​​to generate a subset of the plurality of weight values ​​and a mask value associated with the plurality of weight values. The subset of the plurality of weight values ​​is provided to a matrix multiplication unit. The multiplexing unit generates the subset of the plurality of activation values ​​based on the mask value and provides the subset of the plurality of activation values ​​to the matrix multiplication unit. The matrix multiplication unit performs a set of matrix multiplication operations on the subset of the plurality of weight values ​​and the subset of the plurality of activation values ​​to generate a set of outputs.

Description

Technical Field

[0001] This disclosure relates to computing hardware. More specifically, this disclosure relates to techniques for training and using neural networks.

Background Art

[0002] Neural networks are machine learning models used in a variety of different applications (e.g., image classification, computer vision, natural language processing, speech recognition, handwriting recognition, etc.). Neural networks may be trained for a particular purpose by running a dataset through them, comparing the results from the neural network to known results, and updating the network based on the differences.

Summary of the Invention

[0003] Deep neural networks (DNNs) have grown exponentially in size over the past few years to achieve better accuracy. Despite their high accuracy, DNNs typically require significant computational costs both for training and inference. Different types of techniques can be used to reduce the computational costs associated with DNNs and / or to shrink the size of DNNs. One type of technique is the addition of sparsity techniques. These techniques introduce sparsity into the input data, weights, and / or activations.

Brief Description of the Drawings

[0004] Various embodiments of this disclosure are not limited to the figures in the accompanying drawings, which are illustrated by way of example. [Figure 1] A computing system for sparsifying a narrow data format for a neural network according to some embodiments is shown. [Figure 2]This section shows exemplary training iterations for training neural networks in several embodiments. [Figure 3] Exemplary neural network layers in several embodiments are shown. [Figure 4] The following shows exemplary matrix multiplication layers in several embodiments. [Figure 5] Several embodiments of a neural network layer are shown. [Figure 6] The following shows exemplary matrix multiplication layers in several embodiments. [Figure 7] This document describes a process for sparsifying data in a narrow data format according to several embodiments. [Figure 8] Simplified block diagrams of exemplary computer systems according to several embodiments are shown. [Figure 9] Several embodiments of neural network processing systems are shown. [Modes for carrying out the invention]

[0005] In the following description, numerous examples and specific details are given for illustrative purposes to provide a complete understanding of the Disclosure. Such examples and details should not be construed as unduly limiting the elements of the claims or the subject matter of the claims as a whole. It will be apparent to those skilled in the art, based on the language of the different claims, that the subject matter of the claims may include some or all of the features of these examples, individually or in combination, and may further include modifications and equivalents of the features and techniques described herein.

[0006] This section describes techniques for sparsifying narrow data formats for neural networks. In some embodiments, the computing system provides tools for sparsifying the inputs to each layer of a neural network when training the neural network or when using the neural network for inference. The data used to implement the neural network may be in a narrow data format. A narrow data format may include a floating-point data format that uses a small number of bits (e.g., 1 to 4 bits for the mantissa) to represent some or all of a floating-point number. In contrast, a wide data format may include conventional floating-point formats (e.g., single-precision floating-point format (e.g., FP32 or float32), double-precision floating-point format (e.g., FP64 or float64), half-precision floating-point format (e.g., FP16)) or any floating-point data format that uses a mantissa greater than 4 bits. To sparsify the inputs to the neural network layers, the tool processes a vector of input values ​​in defined size blocks. For each block of input values, the tool determines a defined ratio of values ​​to be removed (e.g., dropped). The remaining input values ​​are used as input to the neural network layers (e.g., to train the neural network, to use the neural network for inference, etc.). The tool can sparsify one or more inputs to each layer of the neural network.

[0007] The technique described herein offers many advantages and benefits over conventional methods of training and using neural networks. For example, by implementing a neural network using a narrow data format and then sparsifying the inputs to the layers of the neural network, the number of cycles required for some of the hardware used to implement the layers of the neural network to perform the computation is reduced. Conventional methods of training and using neural networks require many more cycles to perform the same computation.

[0008] Figure 1 shows a computing system for sparsifying a narrow data format for a neural network according to several embodiments. As shown, the computing system 100 includes a neural network manager 105, a sparsification engine 110, a quantizer 115, and memory devices 120 and 125. The neural network definition memory device 120 is configured to store the definition of the neural network. In some embodiments, the definition of the neural network specifies the number of layers in the neural network, the number of nodes in each layer, the weights in the neural network, the activation function used in the nodes, the loss function used to calculate the loss, and so on.

[0009] The training data storage device 125 stores a training dataset for training a neural network. In some embodiments, the training dataset includes a set of samples (also called training samples). Each sample includes a set of inputs (also called input data) and a set of expected outputs (also called expected output data). For example, the set of inputs for a sample can be a matrix or vector of sets of numbers (e.g., integers, floating-point numbers, etc.). For example, the set of outputs for a sample can be a matrix or vector of sets of numbers (e.g., integers, floating-point numbers, etc.). Furthermore, the training data storage device 125 may store training parameters for training a neural network. Examples of training parameters may include the number of samples in the training dataset, the batch size (e.g., the number of samples processed in a training iteration), the exponential block size (e.g., the number of values ​​sharing an exponent), the sparsity block size (e.g., the number of values ​​in a block processed during sparsification), and a defined ratio of values ​​to retain or drop within sparsity blocks.

[0010] In some embodiments, the storage devices 120 and 125 are implemented in a single physical storage device, while in other embodiments, the storage devices 120 and 125 may be implemented across several physical storage devices. Figure 1 shows the storage devices 120 and 125 as part of the computing system 100, but those skilled in the art will understand that the neural network definition storage device 120 and / or the training data storage device 125 may be located outside the computing system 100 in some embodiments.

[0011] The neural network manager 105, the sparsification engine 110, and the quantizer 115 will be described with reference to several exemplary operations shown in Figures 2 to 6. The first exemplary operation shows how a neural network is trained. For this operation, the neural network manager 105 receives a request from the computing system 100 (or another device such as a client device communicably coupled to the computing system 100) to train the neural network 210 with a training dataset. In response to this request, the neural network manager 105 accesses the neural network definition storage device 125 and reads the neural network definition associated with the neural network 210, along with the training parameters associated with the neural network 210. Next, the neural network manager 105 generates the neural network 210 based on the neural network definition. As shown in Figure 2, the neural network 210 includes layers 215a to n. Each layer 215a to n includes a corresponding set of weights 220. The weights can be represented using a narrow data format. As mentioned above, narrow data formats can include floating-point data formats that use fewer bits (e.g., 1 to 4 bits for the mantissa) to represent some or all of a floating-point number. In this example, the neural network manager 105 initializes the weight values ​​220a to n with randomly generated values ​​(e.g., integers, floating-point numbers, etc.).

[0012] Next, the neural network manager 105 accesses the training data storage device 135 and reads the training dataset specified for training the neural network 210. Here, the neural network manager 105 starts the training phase of the neural network 210. Here, a defined number of training iterations, specified in the training parameters, are performed on the neural network 210. Figure 2 shows exemplary training iterations for training a neural network according to several embodiments. Specifically, Figure 2 shows an example of training iterations performed on the neural network 210 during the training phase of this exemplary operation. For the training iterations, the neural network 210 is trained using batches of samples 200 containing training data samples 205a-m. The number of samples in a batch of samples 200 is determined based on the batch size specified in the training parameters associated with the neural network 210. To generate batches of samples 200, the neural network manager 105 randomly selects samples 205a-m from the training dataset. Next, the neural network manager 105 processes each sample 205 in the batch of samples 200 via the forward pass (also called the feedforward pass) of the neural network 210. The neural network 210 generates an output for each sample 205 processed through it.

[0013] Based on the output that the neural network 210 generates for a batch of samples 200 and the expected output of the training data samples 205a~m within the batch of samples 200, the neural network manager 105 calculates the corresponding loss values ​​230a~m. The neural network 210 can be configured to calculate the loss values ​​230a~m using any number of different loss functions (e.g., mean squared error loss function, mean squared logarithmic error loss function, mean absolute error loss function, cross-entropy loss function, etc.). Based on the batch of loss values ​​225, the neural network manager 105 calculates the average of the loss values ​​230a~m.

[0014] Next, the neural network manager 105 generates a set of gradients during the reverse pass of the neural network 210 using the average of the loss values ​​230a~m. In some embodiments, the neural network manager 105 generates the set of gradients using backpropagation techniques. Then, the neural network manager 105 uses the set of gradients to adjust the weights 220a~n of the neural network 210 in operation 230. In some embodiments, the neural network manager 105 adjusts the weights 220a~n in the neural network 210 using a gradient method (e.g., gradient descent, stochastic gradient descent, etc.).

[0015] Figure 2 shows an example of iterations of the training neural network 210. In some embodiments, the neural network manager 105 trains the neural network 210 using batches of samples randomly selected from a training dataset of the same size as batches of samples 200, through a defined number of iterations specified by training parameters associated with the neural network 210. During this training phase, the weights 220a~n of the neural network are adjusted in the reverse pass of each iteration.

[0016] After the training phase, the neural network 210 can be used for inference. When the neural network 210 is used for inference, the forward path portion of the neural network 210 is utilized. That is, the neural network manager 105 provides the input data to the neural network 210, which processes it through each layer 215a to n. The output generated by layer 215n is the output data associated with the input data. The output data may also be predictions generated by the neural network 210 based on learned values ​​of a set of weights 220a to n.

[0017] Figure 3 shows exemplary neural network layers 300 according to several embodiments. In particular, Figure 3 shows exemplary data flow passing through the neural network layer 300. The exemplary data flow may be a feedforward data flow that occurs during training of the neural network to which the neural network layer 300 belongs or during the use of the neural network for inference. In some embodiments, the neural network layer 300 can be used to implement any of layers 215a to n of the neural network 210.

[0018] As shown in Figure 3, the neural network layer 300 includes a matrix multiplication layer 305, a vector operation layer 310, and a quantization layer 315. The matrix multiplication layer 305 is configured to receive two input matrices, perform a sparsification operation on one or more input matrices, perform a set of matrix multiplication operations (e.g., a set of dot product operations) on the two inputs, and produce an output matrix which is the result of the set of matrix multiplication operations. In some embodiments, a sparsification engine 110 is used to implement the sparsification operation. The vector operation layer 310 receives an input matrix, performs a set of vector operations (e.g., softmax operation, activation operation, normalization operation, etc.) on the input matrix, and produces an output matrix which is the result of the set of vector operations. The quantization layer 315 receives an input matrix, performs a set of quantization operations to degrade the precision of the input matrix values, and produces an output matrix which includes the degraded precision values. In some embodiments, a quantizer 315 is used to implement the quantization operation.

[0019] The exemplary data flow shown in Figure 3 begins with a matrix multiplication layer 305 that receives input values ​​320 and weight values ​​325. Input values ​​320 can be a matrix of activation values ​​received from previous layers in the neural network. For example, if neural network layer 300 is used to implement layer 215c of neural network 210, then input values ​​320 can be a matrix of activation values ​​received from and generated by layer 215b. Weight values ​​325 can be a matrix of weight values ​​contained in neural network layer 300. Continuing with the example where neural network layer 300 is used to implement layer 215c of neural network 210, weight values ​​325 is a matrix containing the set of weight values ​​220c. Upon receiving weight values ​​325, matrix multiplication layer 325 performs a set of sparsification operations to add sparsity to weight values ​​325 (e.g., converting weight values ​​to zero, removing weight values, etc.). Next, the matrix multiplication layer 305 performs a set of matrix multiplication operations on the input values ​​320 and the sparsified weight values ​​325, generating an output matrix which is the result of the set of matrix multiplication operations. The matrix multiplication layer 305 provides the output matrix as input to the vector operation layer 310.

[0020] When the vector operation layer 310 receives the output matrix from the matrix multiplication layer 305, the vector operation layer 310 executes a set of vector operations on the matrix and generates an output matrix that is the result of the set of vector operations. Examples of vector operations include softmax operations, activation operations, normalization operations, and the like. When the set of vector operations is executed, the vector operation layer 310 provides the output matrix as an input to the quantization layer 315. In response to receiving the matrix, the quantization layer 315 executes a set of quantization operations on the matrix and generates an output value 330. The set of quantization operations reduces the precision of the values within the matrix. The output value 330 can be a matrix that includes values of reduced precision represented using a narrow data format. The quantization layer 315 provides the output value 330 as an input to the next layer within the neural network. Continuing with the example where the neural network layer 300 is used to implement layer 215c of the neural network 210, the quantization layer 315 provides the output value 330 as an input to layer 215d.

[0021] FIG. 4 shows an exemplary matrix multiplication layer 400 according to some embodiments. In particular, FIG. 4 shows an example of sparsifying one of two inputs to the matrix multiplication layer and performing matrix multiplication on the two inputs. In some embodiments, the matrix multiplication layer 400 can be used to implement the matrix multiplication layer 305. As shown in FIG. 4, the matrix multiplication layer 400 includes a hardware engine 405 that includes a multiplexing unit 410 and a matrix multiplication unit 415. The multiplexing unit 410 and the matrix multiplication unit 415 can each be realized by a set of circuits.

[0022] As shown in the figure, the matrix multiplication layer 400 receives a first input 420 and a second input 450. In this example, the input 420 is a vector of eight weight values from a matrix of weight values (e.g., weight values 325 in the neural network layer 300) included in a neural network layer (not shown in FIG. 4). The eight weight values are represented using a narrow floating-point data format that uses a lower-bit mantissa (e.g., 1 to 4 bits) and a shared exponent 422 (e.g., a shared 8-bit exponent). Here, the exponent block size, which represents the number of values sharing the exponent, is 8. Thus, all eight weight values of the input 420 share one exponent value. The input 450 is a vector of eight activation values from a matrix of activation values received from and thereby generated by the previous neural network layer. The eight activation values are represented using the same narrow floating-point data format that uses a lower-bit mantissa and a shared exponent 452. As described above, the exponent block size in this example is 8. Thus, all eight activation values of the input 450 share one exponent value.

[0023] When the matrix multiplication layer 400 receives the input 420, the sparsification engine 110 segments the input 420 into blocks based on the sparsity block size specified in the training parameters. In this example, the sparsity block size is 4. Thus, the sparsification engine 110 segments the input 420 into blocks 425 and 430, each containing four weight values ​​from the input 420. In some embodiments, the sparsity block size is a factor of the exponential block size. For example, in this example, the sparsity block size 4 is a factor of the exponential block size 8 (i.e., 8 is equally divisible by 4). Next, the sparsification engine 110 removes weight values ​​from each block 425 and 430 based on a defined ratio of the values ​​to be retained in the sparsity blocks specified in the training parameters, where the defined ratio is 50%. In this example, the sparsification engine 110 determines 50% of the weight values ​​in block 425 with the highest absolute value and generates block 435 containing these weight values ​​(i.e., the sparsification engine 110 drops the other weight values ​​from block 425). Furthermore, the sparsification engine 110 determines 50% of the weight values ​​in block 430 with the highest absolute value and generates block 440 containing those weight values ​​(i.e., the sparsification engine 110 drops the other weight values ​​from block 430). In some embodiments, instead of using absolute values, the sparsification engine 110 can determine which weight values ​​to retain based on any number of different metrics (e.g., entropy, perplexity, etc.) for measuring the importance of the values.

[0024] After removing weight values ​​from blocks 425 and 430, the sparsification engine 110 generates a bitmask 445 containing the mask value for each weight value in the input 420. For weight values ​​that were not removed (i.e., weight values ​​that are in the top 50% of the weight values ​​in each block), the sparsification engine 110 stores a first defined value (1 in this example) as the mask value at the corresponding position in the bitmask 445. For the removed weight values ​​in input 420, the sparsification engine 110 stores a second defined value (0 in this example) as the mask value at the corresponding position in the bitmask 445. For example, if the first weight value in input 420 is in the top 50% of the weight values ​​in block 425, the sparsification engine 110 stores the value 1 at the first position (i.e., the leftmost position) of the bitmask 445. If the first weight value in input 420 is not in the top 50% of the weight values ​​in block 425, the sparsification engine 110 stores the value 0 at the first position in the bitmask 445. The sparsification engine 110 performs this operation for each weight value in the input 420. In some embodiments where the neural network layer implemented by the matrix multiplication layer 400 is used for inference, the bitmask 445 can be pre-calculated (for example, calculated offline before the neural network layer is used for inference). In other embodiments where the neural network layer implemented by the matrix multiplication layer 400 is trained, the bitmask 445 can be learned using gradient descent. Once the bitmask 445 is generated, the sparsification engine 110 provides the bitmask as input to the multiplexing unit 410. Furthermore, the sparsification engine 110 provides blocks 435 and 440 together as input to the matrix multiplication unit 415.

[0025] When the matrix multiplication layer 400 receives input 450, it provides it as input to the multiplexing unit 410. In response to receiving input 450 and bitmask 445, the multiplexing unit 410 determines a subset of activation values ​​in input 450 based on bitmask 445. In this example, the multiplexing unit 410 determines a subset of input 450 by identifying the activation values ​​in input 450 that correspond to mask value 1 in bitmask 445 and including them in the subset. For example, if the first, third, seventh, and eighth mask values ​​in bitmask 445 have a mask value of 1, the multiplexing unit 410 includes the first, third, seventh, and eighth activation values ​​from input 450 in the subset. The multiplexing unit 410 then generates an output 460 containing the determined subset of activation values ​​in input 450 and provides it as input to the matrix multiplication unit 415.

[0026] When the matrix multiplication unit 415 receives blocks 435 and 440 as its first input and output 460 as its second input, it performs a set of matrix multiplication operations on these two inputs to produce an output value 465. For example, the matrix multiplication unit 415 can calculate the dot product between blocks 435 and 440 and output 460 to produce a scalar output of output value 465. Finally, the matrix multiplication unit 415 provides the output value 465 to the next layer of the neural network (e.g., the vector operation layer 310).

[0027] In this example, the matrix multiplication unit 415 is configured to perform the dot product of four elements in one execution cycle. Thus, the matrix multiplication unit 415 can perform the dot product between blocks 435 and 440 (first inputs of four values) and output 460 (second inputs of four values) in one execution cycle. If the matrix multiplication unit 415 performed the dot product between inputs 420 and 450, it would have required two execution cycles (i.e., the first inputs of 420 and 450). ofThis involves a first cycle that calculates the dot product between four values, and a second cycle that calculates the dot product between the last four values ​​in each of the inputs 420 and 450. Thus, using the sparsification technique described above with reference to Figure 4, the number of execution cycles required to determine the product between inputs 420 and 450 is reduced by 50% (i.e., a twofold increase in speed).

[0028] As explained, referring to Figure 4, the operation in the example described above shows how the vector of weight values ​​from the weight value matrix and the vector of activation values ​​from the activation value matrix are processed through the matrix multiplication layer 400. Thus, this operation can be repeated for the remaining vector of weight values ​​in the weight value matrix and the vector of activation values ​​in the activation value matrix.

[0029] Referring to Figure 3, the neural network layer described above includes a layer that performs quantization operations (e.g., quantization layer 315). In some embodiments, the neural network layer may include a layer that performs sparsification operations. Figure 5 shows another exemplary neural network layer 500 according to some embodiments. In particular, Figure 5 shows an example data flow through a neural network layer that includes a layer for performing sparsification operations. The exemplary data flow may be a feedforward data flow that occurs during training of the neural network to which the neural network layer 500 belongs or during the use of the neural network for inference. In some embodiments, the neural network layer 500 can be used to implement any of layers 215a to n of the neural network 210.

[0030] As shown in Figure 5, the neural network layer 500 includes a matrix multiplication layer 505, a vector operation layer 510, and a sparsification layer 515. The matrix multiplication layer 505 can be implemented by the matrix multiplication layer 305, and the vector operation layer 510 can be implemented by the vector operation layer 310. The sparsification and quantization layer 515 is similar to the quantization layer 315, except that the sparsification and quantization layer 515 is configured to perform a set of sparsification operations on the input matrix (performed, for example, by the sparsification engine 110). In some embodiments, a set of quantization operations is performed on the input matrix before a set of sparsification operations is performed on the input matrix. In other embodiments, a set of quantization operations is performed on the input matrix after a set of sparsification operations has been performed on the input matrix. In some embodiments, a set of quantization operations can be performed on the input matrix in combination with a set of sparsification operations.

[0031] The exemplary data flow shown in Figure 5 begins with a matrix multiplication layer 505 that receives input values ​​520 and weight values ​​525. The input values ​​520 could be a matrix of activation values ​​received from previous layers in the neural network. For example, if neural network layer 500 is used to implement layer 215b of neural network 210, then input values ​​520 could be a matrix of activation values ​​received from and generated by layer 215a. The weight values ​​525 could be a matrix of weight values ​​contained in neural network layer 500. Continuing with the example where neural network layer 500 is used to implement layer 215b of neural network 210, weight values ​​525 is a matrix containing the set of weight values ​​220b.

[0032] After receiving the weight values ​​525, the matrix multiplication layer 525 performs a set of sparsification operations to add sparsity to the weight values ​​525 (e.g., converting the weight values ​​to 0, deleting the weight values, etc.). In this example, the matrix multiplication layer 505 also performs a set of sparsification operations to add sparsity to the input values ​​520 (e.g., converting the weight values ​​to 0, deleting the weight values, etc.). The matrix multiplication layer 505 performs a set of matrix multiplication operations on the sparsified input values ​​520 and the sparsified weight values ​​525 to produce an output matrix which is the result of the set of matrix multiplication operations. The matrix multiplication layer 505 provides the output matrix as input to the vector operation layer 510.

[0033] In response to receiving an output matrix from the matrix multiplication layer 505, the vector operation layer 510 performs a set of vector operations on the matrix, producing an output matrix which is the result of the set of vector operations. As mentioned above, some examples of vector operations can include softmax, activation, and normalization operations. After the set of vector operations has been performed, the vector operation layer 510 provides the output matrix as input to the sparsification and quantization layer 515. Upon receiving the matrix, the sparsification and quantization layer 515 performs a set of quantization operations and a set of sparsification operations on the matrix to produce an output value 530. The set of quantization operations reduces the precision of the values ​​in the matrix, and the set of sparsification operations adds sparsity to the matrix (e.g., converting values ​​to 0, deleting values, etc.). The output value 530 can be a matrix containing sparsified and depreciated values ​​represented using a narrow data format. The sparsification and quantization layer 515 provides the output value 530 as input to the next layer in the neural network. Continuing with the example where neural network layer 500 is used to implement layer 215b of neural network 210, sparsification and quantization layer 515 provides output value 530 as input to layer 215c.

[0034] Figure 6 shows exemplary matrix multiplication layers 600 according to several embodiments. Specifically, Figure 6 shows an example in which each input to the matrix multiplication layer is sparsified and matrix multiplication is performed on the two inputs. In some embodiments, the matrix multiplication layer 600 can be used to implement the matrix multiplication layer 505. As shown in Figure 6, the matrix multiplication layer 600 includes a hardware engine 604. The hardware engine 604 includes a mask alignment unit 606, a multiplexing unit 608, a multiplexing unit 610, and a matrix multiplication unit 612. The mask alignment unit 606, the multiplexing unit 608, the multiplexing unit 610, and the matrix multiplication unit 612 can each be implemented by a set of circuits.

[0035] As shown in Figure 6, the matrix multiplication layer 600 receives a first input 615 and a second input 650. In this example, input 615 is a vector of 16 weight values ​​from a matrix (not shown in Figure 6) of weight values ​​contained in the neural network layer (e.g., weight value 525 in neural network layer 500). The 16 weight values ​​are represented using a narrow floating-point data format with a lower mantissa (e.g., 1-4 bits) and a shared exponent 617 (e.g., a shared 8-bit exponent). Here, the exponent block size, which represents the number of values ​​that share the exponent, is 16. Thus, all 16 weight values ​​of input 615 share one exponent value. Input 650 is a vector of 16 activation values ​​from a matrix of activation values ​​received from the previous neural network layer and generated by it. The 16 activation values ​​are represented using the same narrow floating-point data format with a lower mantissa and a shared exponent 652. As mentioned above, the exponent block size in this example is 16. Therefore, all 16 activation values ​​of input 650 share a single exponential value.

[0036] In response to the matrix multiplication layer 600 receiving the input 615, the sparsification engine 110 segments the input 615 into blocks based on the sparsity block size specified in the training parameters. In this example, the sparsity block size is 8. Thus, the sparsification engine 110 segments the input 615 into blocks 620 and 625, each containing 8 weight values ​​from the input 615. In some embodiments, the sparsity block size is a factor of the exponential block size. For example, in this example, the sparsity block size 8 is a factor of the exponential block size 16 (i.e., 16 is equally divisible by 8). Next, the sparsification engine 110 removes weight values ​​from each block 620 and 625 based on a defined ratio of the values ​​to be retained in the sparsity blocks specified in the training parameters. In this example, the defined ratio is 50%. As shown in Figure 6, the sparsification engine 110 determines 50% of the weight values ​​in block 620 with the highest absolute value and generates block 630 containing these weight values ​​(i.e., the sparsification engine 110 drops the other weight values ​​from block 620). Furthermore, the sparsification engine 110 determines 50% of the weight values ​​in block 625 with the highest absolute value and generates block 635 containing those weight values ​​(i.e., the sparsification engine 110 drops the other weight values ​​from block 625). In some embodiments, instead of using absolute values, the sparsification engine 110 can determine which weight values ​​to retain based on any number of different metrics (e.g., entropy, perplexity, etc.) for measuring the importance of the values.

[0037] Once the sparsification engine 110 has finished removing weight values ​​from blocks 620 and 625, it generates a bitmask 640 containing a mask value for each weight value in the input 615. For the weight values ​​that were not removed (i.e., the weight values ​​are included in the top 50% of the weight values ​​in each block), the sparsification engine 110 stores a first defined value (1 in this example) as the mask value at the corresponding position in the bitmask 640. For the removed weight values ​​in the input 615, the sparsification engine 110 stores a second defined value (0 in this example) as the mask value at the corresponding position in the bitmask 640. The sparsification engine 110 repeats this operation for each weight value in the input 615. In some embodiments where the neural network layer implemented by the matrix multiplication layer 600 is used for inference, the bitmask 640 can be pre-calculated (for example, calculated offline before the neural network layer is used for inference). In other embodiments in which the neural network layers implemented by the matrix multiplication layer 600 are trained, the bitmask 640 can be learned using gradient descent. After generating the bitmask 640, the sparsification engine 110 provides it as input to the mask alignment unit 606. Furthermore, the sparsification engine 110 provides both blocks 630 and 635 as input to the multiplexing unit 608.

[0038] When the matrix multiplication layer 600 receives the input 650, the sparsification engine 110 segments the input 650 into blocks based on a sparsity block size of 8 in this example. Thus, the sparsification engine 110 segments the input 650 into blocks 655 and 660, each containing 8 activation values ​​from the input 650. Next, the sparsification engine 110 removes activation values ​​from each block 655 and 650 based on a defined ratio of values ​​to be retained in the sparsity blocks, specified by the training parameters, which is 50% in this example. Thus, the sparsification engine 110 determines 50% of the activation values ​​in block 655 with the highest absolute value and generates block 670 containing those activation values ​​(i.e., the sparsification engine 110 drops the other activation values ​​from block 655). The sparsification engine 110 also determines 50% of the activation values ​​in block 660 with the highest absolute values ​​and generates block 675 containing the activation values ​​(i.e., the sparsification engine 110 drops the other activation values ​​from block 660). In some embodiments, instead of using absolute values, the sparsification engine 110 can determine which activation values ​​to retain based on any number of different metrics (e.g., entropy, perplexity, etc.) to measure the importance of the values.

[0039] After completing the removal of activation values ​​from blocks 655 and 660, the sparsification engine 110 generates a bitmask 665 containing the mask values ​​for each activation value in the input 650. For the activation values ​​that were not removed (i.e., the activation values ​​are included in the top 50% of the activation values ​​in each block), the sparsification engine 110 stores a first defined value (1 in this example) as the mask value at the corresponding position in the bitmask 665. For the removed activation values ​​in the input 650, the sparsification engine 110 stores a second defined value (0 in this example) as the mask value at the corresponding position in the bitmask 665. The sparsification engine 110 performs this operation for each activation value in the input 650. In some embodiments where the neural network layer implemented by the matrix multiplication layer 600 is used for inference, the bitmask 665 can be computed on the fly (as opposed to a bitmask 640 that may be computed in advance). In other embodiments where the neural network layer implemented by the matrix multiplication layer 600 is trained, the bitmask 665 can be learned using gradient descent. When the sparsification engine 110 generates the bitmask 665, it provides it as input to the mask alignment unit 606. Furthermore, the sparsification engine 110 provides both blocks 670 and 675 as input to the multiplexing unit 610.

[0040] In response to receiving bitmasks 640 and 665, the mask alignment unit 606 generates an alignment mask by performing element-wise multiplication of bitmasks 640 and 665. The alignment mask contains the product of each pair of corresponding elements in bitmasks 640 and 665. For example, the first element in the alignment mask stores the product between the first element in bitmask 640 and the first element in bitmask 665, the second element stores the product between the second element in bitmask 640 and the second element in bitmask 665, the third element stores the product between the third element in bitmask 640 and the third element in bitmask 665, and so on. In some embodiments, an objective function (e.g., a loss function) can be used to maximize the overlap of initially defined values ​​in bitmasks 640 and 665 during training. Next, the mask alignment unit 606 sends the alignment mask and bit mask 640 to the multiplexing unit 608. The mask alignment unit 606 also sends the alignment mask and bit mask 665 to the multiplexing unit 610.

[0041] When the multiplexing unit 608 receives the alignment mask, bit mask 640, blocks 630 and 635, it determines a subset of weight values ​​in blocks 630 and 635 based on the alignment mask and bit mask 640. To determine which weight values ​​in blocks 630 and 635 should be included in the subset of blocks 630 and 635, the multiplexing unit 608 iterates through the mask values ​​in bit mask 640. If a mask value stores a value of 1, the multiplexing unit 608 looks at the same position in the alignment mask. If the mask value at the position also stores a value of 1, the multiplexing unit 608 includes that weight value in the subset. Otherwise, the multiplexing unit 608 continues iterating through bit mask 640 in the same way. Here, the multiplexing unit 608 determines the first weight value identified in this way in the subset. ofIt includes four weight values. In some embodiments, the multiplexing unit 608 includes all the weight values ​​identified in the subset. The multiplexing unit 608 then generates an output 645 containing the subset of weight values ​​in blocks 630 and 635 and provides it as input to the matrix multiplication unit 612. The multiplexing unit 610 uses the same technique to determine a subset of activation values ​​in blocks 670 and 675 based on the alignment mask and bit mask 665. The multiplexing unit 610 generates an output 680 containing the subset of activation values ​​in blocks 670 and 675 and provides it as input to the matrix multiplication unit 612. In some embodiments, the multiplexing units 608 and 610 employ a row-sharing technique in which multiple subsets of values ​​at the same position (e.g., each subset of weight values ​​contains the values ​​at input positions 2, 3, 4, 6, 8, 11, 12, and 16) are iterated over together (e.g., blocks 630 and 635 of the multiplexing unit 608, and blocks 670 and 675 of the multiplexing unit 610). In this way, the cost of multiplexing is reduced because the same index is used to process these multiple subsets of values ​​and can therefore be reused to process each subset of the value.

[0042] When the matrix multiplication unit 612 receives outputs 645 and 680, it performs a set of matrix multiplication operations on these two inputs to produce output value 685. For example, the matrix multiplication unit 612 can calculate the dot product between block output 645 and output 670 to produce a scalar output of output value 685. The matrix multiplication unit 612 then provides output value 665 to the next layer of the neural network layer (e.g., the vector operation layer 510).

[0043] In this example, the matrix multiplication unit 612 is configured to perform the dot product of four elements in one execution cycle. Thus, the matrix multiplication unit 612 can perform the dot product between output 645 (the first input of the four values) and output 680 (the second input of the four values) in one execution cycle. In some embodiments in which the multiplexing units 608 and 610 include all weight values ​​identified in their respective subsets, the matrix multiplication unit 612 can perform the dot product between output 645 (the first input of the four values) and output 680 (the second input of the four values) in one and two execution cycles. If the matrix multiplication unit 612 were to perform the dot product between inputs 615 and 650, it would require four execution cycles (i.e., the first input in each of the inputs 615 and 650). of The first cycle calculates the dot product between the four values, the second in each input 615 and 650 of The second cycle calculates the dot product between the four values, the third in each input 615 and 650 of The third cycle calculates the dot product between the four values, and the second cycle calculates the dot product between the last four values ​​in each input 615 and 650. 4 Sa (cycle). Therefore, using the sparsification technique described above with reference to Figure 6, the number of execution cycles required to determine the product between inputs 620 and 650 is reduced by 50-75% (i.e., a 2-4 times speed increase).

[0044] Referring to Figure 6, the exemplary operation described above illustrates how the vector of weight values ​​from the weight value matrix and the vector of activation values ​​from the activation value matrix are processed through the matrix multiplication layer 600. Thus, this operation can be repeated for the remaining vector of weight values ​​in the weight value matrix and the vector of activation values ​​in the activation value matrix.

[0045] Figure 7 shows a process 700 for sparsifying data in a narrow data format according to several embodiments. In some embodiments, a computing system 100 performs the process 700. The process 700 begins by providing a plurality of activation values ​​received from the first layer of a neural network to a multiplexing unit configured to implement the second layer of a neural network (710). The plurality of activation values ​​are represented using a first plurality of lower-bit mantissa values ​​and a first shared exponent value. Referring to Figure 4 as an example, when the matrix multiplication layer 400 receives input 450, the matrix multiplication layer 400 provides it as input to the multiplexing unit 410.

[0046] Next, in process 700, process 720 performs a set of sparsification operations on multiple weight values ​​included in the second layer of the neural network to generate a subset of the multiple weight values ​​and multiple mask values ​​associated with the multiple weight values. The multiple weight values ​​are represented using a first set of lower bit mantissa values ​​and a second shared exponent value. Referring to Figures 1 and 4 as an example, the sparsification engine 110 performs a set of sparsification operations on the input 420 to generate blocks 435 and 440 and bitmask 445.

[0047] Finally, process 700 provides a subset of weight values ​​to a matrix multiplication unit configured to implement the second layer in the neural network at 730. The multiplexing unit generates a subset of activation values ​​based on a subset of mask values ​​and provides this subset of activation values ​​to the matrix multiplication unit. The matrix multiplication unit is configured to perform a set of matrix multiplication operations on the subsets of weight values ​​and the subsets of activation values ​​to produce a set of outputs. Referring to Figures 1 and 4 as an example, the sparsification engine 110 provides blocks 435 and 440 as inputs to the matrix multiplication unit 415. The multiplexing unit 410 produces an output 460, which is a subset of the input 450. The matrix multiplication unit 415 performs a set of matrix multiplication operations on the output 460 and blocks 435 and 440 to produce an output value 465.

[0048] The techniques described above can be implemented in a wide range of computer systems configured to process neural networks. Figure 8 shows a simplified block diagram of an exemplary computer system 800 that may be used to implement the techniques described above. As shown in Figure 8, the computer system 800 includes one or more processors 802 that communicate with a number of peripheral devices via a bus subsystem 804. These peripheral devices may include a storage subsystem 806 (e.g., including a memory subsystem 808 and a file storage subsystem 810) and a network interface subsystem 816. Some computer systems may further include user interface input devices 812 and / or user interface output devices 814.

[0049] The bus subsystem 804 can provide a mechanism that enables various components and subsystems of the computing system 800 to communicate with each other as intended. Although the bus subsystem 804 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses.

[0050] The network interface subsystem 816 can function as an interface for communicating data between the computer system 800 and other computer systems or networks. Embodiments of the network interface subsystem 816 may include, for example, Ethernet, Wi-Fi and / or cellular adapters, modems (telephone, satellite, cable, ISDN, etc.), digital subscriber line (DSL) units and / or similar.

[0051] The storage subsystem 806 includes a memory subsystem 808 and a file / disk storage subsystem 810. Subsystems 808 and 810, as well as other memories described herein, are examples of non-temporary computer-readable storage media capable of storing executable program code and / or data that provide the functionality of embodiments of the present disclosure.

[0052] The memory subsystem 808 includes a number of memories, including a main random access memory (RAM) 818 for storing instructions and data during program execution and a read-only memory (ROM) 820 for storing fixed instructions. The file storage subsystem 810 can provide persistent (e.g., non-volatile) storage for program and data files and may include optical drives, removable flash memory-based drives or cards, and / or other types of storage media known in the art, along with magnetic or solid-state hard disk drives, associated removable media (such as CD-ROMs, DVDs, and Blu-rays).

[0053] Computer system 800 is illustrative, and it should be understood that many other configurations are possible, some with more or fewer components than system 800.

[0054] Figure 9 shows a neural network processing system according to several embodiments. In various embodiments, the neural network according to this disclosure can be implemented and trained in a hardware environment including one or more neural network processors. The neural network processor can refer to, for example, various graphics processing units (GPUs) (e.g., GPUs from NVIDIA Corp. for processing neural networks), field-programmable gate arrays (FPGAs) (e.g., FPGAs from Xilinx for processing neural networks), or various application-specific integrated circuits (ASICs) or neural network processors including hardware architectures optimized for neural network computation. In this exemplary environment, one or more servers 902, which may include the architecture shown in Figure 8 above, can be coupled to a plurality of control units 910(1) to 910(M) via a communication network 901 (e.g., switches, routers, etc.). The control units 910(1) to 910(M) may also include the architecture shown in Figure 8 above. Each control unit 910(1) to 910(M) can be coupled to one or more NN processors, such as processors 911(1) to 911(N) and 912(1) to 912(N). NN processors 911(1) to 911(N) and 912(1) to 912(N) may include various configurations of functional processing blocks and memory optimized for neural network processing such as training or inference. NN processors are optimized for neural network computation. Server 902 can set NN models and input data to the models in control unit 910, for example, which can be loaded and executed in parallel by NN processors 911(1) to 911(N) and 912(1) to 912(N). Models may include layers and associated weights, for example, as described above. NN processors can load models, apply inputs, and generate output results. NN processors may also implement training algorithms, for example, those described herein.

[0055] <Further exemplary embodiments> In various embodiments, this disclosure includes systems, methods, and apparatus for sparsifying narrow data formats for neural networks. The techniques described herein can be embodied in a machine-readable medium that stores a program executable by a computer system, the program including a set of instructions for performing the techniques described herein. In some embodiments, the system includes a set of processing units and a machine-readable medium that, when executed by at least one processing unit in the set of processing units, stores instructions causing at least one processing unit to perform the techniques described herein. In some embodiments, the machine-readable medium may be, for example, a memory that can be coupled to one or more control units or one or more artificial intelligence processors.

[0056] The following techniques may be implemented individually or in different combinations, or further implemented in conjunction with other techniques described herein.

[0057] For example, in one embodiment, the disclosure includes a machine-readable medium for storing a program executable by at least one processing unit of the apparatus. The program is Multiple activation values ​​received from the first layer of the neural network are provided to a multiplexing unit configured to implement the second layer of the neural network, and the multiple activation values ​​are represented using a first set of lower-bit mantissa values ​​and a first shared exponent value. A set of sparsification operations is performed on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponential value. A subset of the aforementioned weight values ​​is provided to a matrix multiplication unit configured to implement the second layer in the neural network. Includes a set of instructions for The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values.

[0058] In one embodiment, the step of performing the set of sparsifications is: The steps include dividing the plurality of weight values ​​into groups of weight values ​​based on a defined group size, For each group of weight values ​​within the group of weight values, the step of determining the defined ratio of the weight value having the highest value within that group of weight values, Includes, The subset of weight values ​​is generated by, for each group of weight values ​​within the group of weight values, including in the subset of weight values ​​a determined ratio of weight values. The plurality of mask values ​​related to the plurality of weight values ​​are, for each of the plurality of weight values, If the weight value is included in a subset of the plurality of weight values, the first defined value is stored as the mask value in the plurality of mask values ​​associated with the weight value. If the weight value is not included in the subset of the plurality of weight values, the second defined value is stored as the mask value in the plurality of mask values ​​associated with the weight value. It is generated by [the following].

[0059] In one embodiment, the disclosure further includes performing a set of vector operations on the set of outputs.

[0060] In one embodiment, the disclosure further includes, after performing the set of vector operations, performing a set of quantization operations on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs.

[0061] In one embodiment, the disclosure further includes transmitting the quantized set of outputs as input to a third layer of the neural network.

[0062] In one embodiment, the disclosure further includes performing a set of sparsification operations on the quantized set of outputs after performing the set of quantization operations to add sparsity to the quantized set of outputs.

[0063] In one embodiment, the disclosure further includes performing a set of sparsification operations on the set of outputs before performing the set of quantization operations to add sparsity to the set of outputs.

[0064] In one embodiment, the disclosure further includes performing a set of quantization operations on the set of outputs generated by the matrix multiplication unit before performing the set of vector operations to reduce the precision of the set of outputs.

[0065] In one embodiment, each lower bit manzite value of the first plurality of lower bit manzite values ​​includes 4 bits or less, and each lower bit manzite value of the second plurality of lower bit manzite values ​​includes 4 bits or less.

[0066] The above description illustrates various embodiments of the Disclosure, along with examples of how aspects of a particular embodiment may be carried out. These examples should not be considered sole embodiments, but are provided to illustrate the flexibility and advantages of the specific embodiments defined by the following claims. Based on the above disclosure and the following claims, other configurations, embodiments, forms, and equivalents can be adopted without departing from the scope of the Disclosure as defined by the claims.

Claims

1. A machine-readable medium storing a program executable by at least one processing unit of the device, wherein the program is: Multiple activation values ​​received from the first layer of the neural network are provided to a multiplexing unit configured to implement the second layer of the neural network, and the multiple activation values ​​are represented using a first set of lower-bit mantissa values ​​and a first shared exponent value. A set of sparsification operations is performed on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponent value. A subset of the aforementioned weight values ​​is provided to a matrix multiplication unit configured to implement the second layer in the neural network. Includes a set of instructions for The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the aforementioned plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values. The aforementioned program, A set of vector operations is performed on the aforementioned set of outputs. After performing the set of vector operations, a set of quantization operations is performed on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs. After performing the set of quantization operations, a set of sparsification operations is performed on the quantized set of outputs to add sparsity to the quantized set of outputs. A machine-readable medium further including a set of instructions for this purpose.

2. A machine-readable medium storing a program executable by at least one processing unit of an apparatus, wherein the program is: Multiple activation values ​​received from the first layer of the neural network are provided to a multiplexing unit configured to implement the second layer of the neural network, and the multiple activation values ​​are represented using a first set of lower-bit mantissa values ​​and a first shared exponent value. A set of sparsification operations is performed on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponent value. A subset of the aforementioned weight values ​​is provided to a matrix multiplication unit configured to implement the second layer in the neural network. Includes a set of instructions for The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the aforementioned plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values. The aforementioned program, A set of vector operations is performed on the aforementioned set of outputs. After performing the set of vector operations, a set of quantization operations is performed on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs. Before performing the set of quantization operations, a set of sparsification operations is performed on the set of outputs to add sparsity to the set of outputs. A machine-readable medium further including a set of instructions for this purpose.

3. A machine-readable medium storing a program executable by at least one processing unit of an apparatus, wherein the program is: Multiple activation values ​​received from the first layer of the neural network are provided to a multiplexing unit configured to implement the second layer of the neural network, and the multiple activation values ​​are represented using a first set of lower-bit mantissa values ​​and a first shared exponent value. A set of sparsification operations is performed on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponent value. A subset of the aforementioned weight values ​​is provided to a matrix multiplication unit configured to implement the second layer in the neural network. Includes a set of instructions for The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the aforementioned plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values. The aforementioned program, A set of vector operations is performed on the aforementioned set of outputs. Before executing the set of vector operations, a set of quantization operations is performed on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs. A machine-readable medium further including a set of instructions for this purpose.

4. Performing the aforementioned set of sparsification operations means Based on the defined group size, the plurality of weight values ​​are divided into groups of weight values, For each group of weight values ​​within the aforementioned group of weight values, the defined ratio of the weight value having the highest value within that group of weight values ​​is determined. Includes, The subset of weight values ​​is generated by, for each group of weight values ​​within the group of weight values, including in the subset of weight values ​​a determined ratio of weight values. The plurality of mask values ​​related to the plurality of weight values ​​are, for each of the plurality of weight values, If the weight value is included in a subset of the plurality of weight values, the first defined value is stored as the mask value in the plurality of mask values ​​associated with the weight value. If the weight value is not included in the subset of the plurality of weight values, the second defined value is stored as the mask value in the plurality of mask values ​​associated with the weight value. A machine-readable medium according to any one of claims 1 to 3, which is generated by the means described herein.

5. The machine-readable medium according to claim 1 or 2, further comprising a set of instructions for transmitting the quantized set of outputs as input to the third layer of the neural network.

6. The machine-readable medium according to any one of claims 1 to 3, wherein each lower bit mantissa of the first plurality of lower bit mantissas includes 4 bits or less, and each lower bit mantissa of the second plurality of lower bit mantissas includes 4 bits or less.

7. It is a method, A step of providing a plurality of activation values ​​received from a first layer in a neural network to a multiplexing unit configured to implement a second layer in the neural network, wherein the plurality of activation values ​​are represented using a first plurality of lower bit mantissa values ​​and a first shared exponent value. A step of performing a set of sparsification operations on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponent value. The steps include providing a subset of the plurality of weight values ​​to a matrix multiplication unit configured to implement the second layer in the neural network, Includes, The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the aforementioned plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values. The aforementioned method, The steps include: performing a set of vector operations on the set of outputs; After the step of performing the set of vector operations, the step of performing a set of quantization operations on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs, The steps include: after performing the set of quantization operations, performing a set of sparsification operations on the quantized set of outputs to add sparsity to the quantized set of outputs; Methods that further include the above.

8. A method, A step of providing a plurality of activation values ​​received from a first layer in a neural network to a multiplexing unit configured to implement a second layer in the neural network, wherein the plurality of activation values ​​are represented using a first plurality of lower bit mantissa values ​​and a first shared exponent value. A step of performing a set of sparsification operations on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponent value. The steps include providing a subset of the plurality of weight values ​​to a matrix multiplication unit configured to implement the second layer in the neural network, Includes, The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the aforementioned plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values. The aforementioned method, The steps include: performing a set of vector operations on the set of outputs; After the step of performing the set of vector operations, the step of performing a set of quantization operations on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs, Prior to the step of performing the set of quantization operations, the step of performing a set of sparsification operations on the set of outputs to add sparsity to the set of outputs, A method that further includes this.

9. A method, A step of providing a plurality of activation values ​​received from a first layer in a neural network to a multiplexing unit configured to implement a second layer in the neural network, wherein the plurality of activation values ​​are represented using a first plurality of lower bit mantissa values ​​and a first shared exponent value. A step of performing a set of sparsification operations on a plurality of weight values ​​included in the second layer of the neural network to generate a subset of the plurality of weight values ​​and a plurality of mask values ​​associated with the plurality of weight values, wherein the plurality of weight values ​​are represented using a first plurality of lower bit mantissa values ​​and a second shared exponent value. The steps include providing a subset of the plurality of weight values ​​to a matrix multiplication unit configured to implement the second layer in the neural network, Includes, The multiplexing unit is further configured to generate a subset of the plurality of activation values ​​based on the plurality of mask values ​​and to provide the subset of the plurality of activation values ​​to the matrix multiplication unit, The matrix multiplication unit is configured to perform a set of matrix multiplication operations on a subset of the plurality of weight values ​​and a subset of the plurality of activation values ​​to generate a set of outputs. Performing a set of sparsification operations on the aforementioned plurality of weight values ​​reduces the number of cycles the matrix multiplication unit performs to generate the product of the plurality of activation values ​​and the plurality of weight values. The aforementioned method, The steps include: performing a set of vector operations on the set of outputs; Prior to the step of performing the set of vector operations, the step of performing a set of quantization operations on the set of outputs generated by the matrix multiplication unit to reduce the precision of the set of outputs, A method that further includes this.

10. The step of performing the set of sparsification operations is: The steps include dividing the plurality of weight values ​​into groups of weight values ​​based on a defined group size, For each group of weight values ​​within the group of weight values, the step of determining the defined ratio of the weight value having the highest value within that group of weight values, Includes, The subset of weight values ​​is generated by, for each group of weight values ​​within the group of weight values, including in the subset of weight values ​​a determined ratio of weight values. The plurality of mask values ​​related to the plurality of weight values ​​are, for each of the plurality of weight values, If the weight value is included in a subset of the plurality of weight values, the first defined value is stored as the mask value in the plurality of mask values ​​associated with the weight value. If the weight value is not included in the subset of the plurality of weight values, the second defined value is stored as the mask value in the plurality of mask values ​​associated with the weight value. The method according to any one of claims 7 to 9, which is produced by...

11. The method according to claim 7 or 8, further comprising the step of transmitting the quantized set of outputs as input to the third layer of the neural network.

Images