Continuous Sparsity Pattern Neural Network

Abstract

Methods, systems and apparatus for using a neural network with a continuous sparsity pattern, including a computer program encoded on a computer storage medium. One of the methods includes storing a first parameter matrix of a neural network with a continuous sparsity pattern in a storage associated with a computing device. The computing device performs an inference process of the neural network to generate an output vector, including reading one or more activation values ​​from an input vector from the storage associated with the computing device, reading a block of non-zero parameter values ​​from the storage associated with the computing device, and multiplying each of the one or more activation values ​​by one or more of the blocks of non-zero parameter values.

Description

Technical Field

[0001] This specification relates to techniques for implementing neural networks on processing hardware. Background Art

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear computational units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. During an inference pass (i.e., the processing performed by the neural network as a whole to generate a set of outputs at a certain time step based on the corresponding set of inputs to the neural network), the output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer).

[0003] Each layer of the network generates the output of the next layer based on the current values ​​of the corresponding parameter set of the layer and one or more transformation operations. The transformation operations of each layer can be performed by a processing device running an installed software kernel that implements the transformation operations of the layer. Therefore, a layer described as performing an operation means that the processing device that implements the transformation operations of the layer performs the transformation operations.

[0004] Naturally, as the size and complexity of a neural network increases, the time required to perform an inference pass through the network will also increase. In general, the speed of inference of a neural network is proportional to the number of parameters in the model. This can create a significant processing bottleneck that can limit the types of neural networks that can be implemented on certain types of processing hardware.

[0005] This is particularly challenging for neural networks that have significant throughput requirements, significant latency requirements, or both. For example, real-time audio generation neural networks present significant computational challenges due to the fundamentally high-throughput nature of generating raw audio waveforms. Realistic audio waveform generation typically requires generating thousands of samples per second, such as 24,000 samples per second, which may require performing 24,000 inference passes per second, for example. These performance requirements make it difficult to implement high-throughput neural networks directly on consumer hardware (such as mobile phones and tablet computers), which lacks the types of high-performance or parallel processing hardware available in data centers. As another example, neural networks that operate for sensing or controlling autonomous and semi-autonomous vehicles have significant latency requirements. In other words, neural networks used to perform sensing and control under real-world driving conditions must be fast enough so that the vehicle can react reliably within some well-defined time period (e.g., within microseconds or milliseconds).

[0006] One way to increase inference speed is to generate layers with sparse connections. In this case, a model with sparse connections between layers means that a fairly large number of possible connections are deactivated by having a value of zero. For example, between two layers with 1000 neurons each, there are 1000x1000 possible connections, or 1 million possible connections. Instead of having 1 million connections, the connections can be reduced to a small fraction of the possible connections, for example, only 10,000 or 50,000 connections.

[0007] Inference speed can be further improved by training the model to have block sparsity in the connections. Block sparsity imposes some structure on the sparsity pattern of the connections by computing a single representative value (such as the mean or maximum value) for a square region of the parameter matrix. Block sparsity does impose some structure in the model parameters, but the pattern of squares in the matrix has a basically uniform randomness. Summary of the invention

[0008] This specification describes techniques for increasing the inference speed of a neural network and reducing the data storage requirements of a neural network by arranging parameter values ​​into a continuous sparsity pattern. Training a model to have continuous sparsity involves imposing a structured sparsity pattern in which a significant number of non-zero values ​​may be continuous along the same dimension in a parameter matrix. This also means that a significant number of zero values ​​may be continuous along the same dimension in a parameter matrix.

[0009] It was found experimentally that neural networks with continuous sparsity perform roughly the same as those with unstructured sparsity (i.e., the same sparsity, but with non-zero values ​​randomly distributed), but crucially, the sparsity pattern can be stored with reduced data storage capacity. For example, if parameter values ​​are in 1xm blocks (where m is an integer greater than 1), the data required to store the locations of non-zero values ​​is reduced by a factor of m compared to unstructured sparsity.

[0010] Furthermore, as explained below, continuous sparsity allows neural networks to be implemented with more efficient encoding than unstructured sparsity, thereby reducing the number of computational operations required to implement them.

[0011] Both advantages are critical in situations where, although the neural network may be created on a server, it is utilized by client devices such as mobile phones that have low memory capacity and limited computing power. Note that the computing power and memory bandwidth on current mobile CPUs (central processing units) are three orders of magnitude and two orders of magnitude smaller, respectively, than on GPUs (graphics processing units).

[0012] What is considered a sufficiently large number of consecutive zero or non-zero parameter values ​​depends on the underlying training set and design goals of the model. For example, the system can train a model based on hyperparameters that specify one or more of these constraints (e.g., the likelihood of having blocks of consecutive non-zero parameters of at least length l with at least frequency f within the parameter matrix).

[0013] For simplicity of representation, this specification will use the term "sparse" to refer to a (two-dimensional) pattern of values ​​(i.e., a matrix) in which the number of zero values ​​is greater than the number of non-zero values. Specifically, the proportion of zero values ​​may be at least 80%, at least 90%, or at least 95% (such as at least 96.4% or at least 97.8%), or even at least 99%. The term "sparseness pattern" refers to a data set that indicates where the non-zero values ​​of a parameter matrix are.

[0014] The term "block" is used to refer to a (non-square) rectangle of non-zero values ​​in a pattern. In general, a block cannot be extended in any dimension without including at least one zero value of the sparsity pattern; thus, a block is the largest contiguous block of non-zero values ​​that can be formed. However, it is also possible for two or more such blocks to be adjacent.

[0015] We will use the term "continuous sparsity" to refer to any sparsity pattern that consists essentially of multiple zero values, and the blocks of non-zero values ​​have a block size where one dimension is much larger than another dimension (e.g., at least 4 times larger (e.g., exactly 4 times larger), or even at least 16 times larger (e.g., exactly 16 times larger) than the other dimension), and the number of zero values ​​is greater than the number of non-zero values. For example, each of the blocks can have the same size; that is, the blocks are nxm blocks of non-zero values ​​(where n and m are integers), and n is different from m.

[0016] A parameter matrix with a continuous sparsity pattern may have a column sparsity where non-zero parameter values ​​are continuous along the columns of the parameter matrix (e.g., in the case of equal-sized blocks, n is greater than m), or may have a row sparsity where non-zero parameter values ​​are continuous along the rows of the parameter matrix (e.g., in the case of equal-sized blocks, m is greater than n). Note that in the case of row sparsity, there may be more than one block in each row of the parameter matrix (e.g., multiple blocks are separated by zero values); similarly, in the case of column sparsity, there may be more than one block in each column of the parameter matrix (e.g., multiple blocks are separated by zero values). In some parameter matrices, two or more blocks may be adjacent. Furthermore, in the case of row sparsity, multiple columns may each contain multiple non-zero values; and in the case of column sparsity, multiple rows may each contain multiple non-zero values.

[0017] Here, it is understood that the horizontal (row) direction in the parameter matrix (i.e., the direction in which the blocks have a range of m) represents different inputs of the layer, while the vertical (column) direction in the parameter matrix (i.e., the direction in which the blocks have a range of n) corresponds to different outputs of the layer (or different neurons of the layer). For example, in one case, the layer performs a multiplication between the parameter matrix and a vector x representing the input to the layer (in other words, in an operation such as Rx, the parameter matrix R of the network parameters is multiplied by the vector x representing the input values), and then performs a function on each of the results of the multiplication. In the case of row sparsity, the process of generating a given output value may include a multiplication process for each block of the corresponding row (where each input value is multiplied by the parameter value of the block and then the results are summed), summing the results over multiple blocks of the row, and performing a nonlinear function on the summed results. The multiplication process for each block of non-zero parameter values ​​may be performed after the block of non-zero parameter values ​​is taken from the storage reading device, and is completed before any other parameter values ​​are read from the memory.

[0018] For example, a continuous sparsity pattern may have blocks where one of the dimensions is 1, such as having dimensions 1 x m or n x 1. However, any n x m block size may be used, such that n is much larger (eg, four times or sixteen times) than m, or vice versa.

[0019] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. The use of continuous sparsity patterns allows neural networks with very important throughput requirements, very important latency requirements, or both to be implemented on processing hardware (e.g., consumer hardware, such as mobile phones (or other mobile devices) and tablet computers) that has only a CPU and less powerful onboard processing devices for controlling automated vehicle functions. In these computing environments, neural networks can be implemented locally on hardware, meaning that the network can run without a network connection to a data center, without the need to use dedicated parallel processing hardware (such as a GPU), and without the need for dedicated hardware. For example, this allows real-time audio waveform generation to be performed on handheld mobile devices, for example, for text-to-speech applications. And it also allows more sophisticated and complex neural networks to be implemented in environments with important latency requirements. Another application of neural networks is the automatic generation of images. Another application of neural networks is the generation of text or any other suitable one-dimensional data stream. A further application of neural networks will be to control agents (e.g., mechanical devices) in environments such as real-world environments based on network inputs that encode information received by sensors and characterize the environment. In one application, the agent can be a transportation device, such as an autonomous or semi-autonomous vehicle.

[0020] If the input to the neural network is text, then the input to the neural network at continuous time may include consecutive characters of the text segment.

[0021] Alternatively, in other applications, the input to the neural network at continuous times may include data representing the environment (e.g., audio data) obtained by sensors at corresponding continuous times.

[0022] In one example, the neural network can be a recurrent neural network (RNN) in which a plurality of neurons each generate an output at each time step (e.g., in each inference pass) that depends on the state of the neuron at the previous time step (which in turn depends on the input(s) to the neuron in the previous inference pass). For example, a layer of the neural network can receive among its inputs at a given time step the output of the neural layer at at least one previous time step (e.g., including the immediately previous time step). BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Included is a visualization of the parameter matrix showing the structure of the continuous sparsity pattern compared to the random sparsity pattern.

[0024] Figure 2 is a diagram of an example system 200 using a continuous sparsity mode.

[0025] Figure 3 is a flow chart of an example process for training a neural network to have a parameter matrix with a continuous sparsity pattern.

[0026] Figure 4 is a flow chart of an example process for performing an inference pass on a neural network with a parameter matrix that has a continuous sparsity pattern.

[0027] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0028] Figure 1 Included is a visualization of the parameter matrix that shows the structure of the continuous sparsity pattern compared to the random sparsity pattern. Figure 1 , black areas represent zero parameter values, and white areas represent non-zero parameter values.

[0029] The random sparsity parameter matrix 120 has a random sparsity pattern. Therefore, the non-zero parameter values ​​are substantially uniformly distributed in the matrix.

[0030] The continuous sparsity parameter matrix 110 has a continuous sparsity pattern. Therefore, a substantial number of non-zero values ​​are continuous along the rows of the parameter matrix.

[0031] Continuous sparsity can be achieved during training by periodically sorting blocks of parameters according to magnitude, and zeroing all parameters of blocks whose parameters meet a criterion indicating that the parameters are small. For example, the sum of the magnitudes of the parameters of a block can be below a certain threshold. The result is a sparsity pattern that has more structure than a random sparsity pattern, and the sparsity pattern improves inference speed relative to a block sparsity pattern.

[0032] Continuous sparsity can be used to increase the inference speed of neural networks in a number of ways.

[0033] Figure 2 is a diagram of an example system 200 using a continuous sparsity pattern. System 200 is an example of a system that can implement the techniques described in this specification. In one application, system 200 can implement a signal generating neural network on parallel processing hardware, such as a neural network that generates audio, images, text, or any other suitable one-dimensional data stream. In doing so, some matrix operations can be performed in parallel by all available processing units, while other matrix operations can be performed in parallel by a subset of the available processing units.

[0034] System 200 includes a neural network system 210 defined by a plurality of layers. In general, each layer has its own parameter matrix. Thus, layer N has a parameter matrix separate from layer N+1. Each layer is implemented by its own computing system. For simplicity, only two such layers are shown as being implemented by two respective computing systems, layer N computing system 220 and layer N+1 computing system 250. Each of these computing systems can be implemented using any suitable combination of general or special computing hardware. For example, each computing system can be implemented using a CPU, a GPU, or a dedicated integrated circuit (e.g., a machine learning accelerator). Computing systems 220 and 250 can be implemented by separate computing hardware or the same computing hardware.

[0035] The layer N computing system 220 includes an activation engine 222 configured to receive an input 205 (e.g., an input vector). The components of the input vector are referred to herein as "activation values". In the case where layer N is the first layer of the network, the activation value is the input to the neural network. In an application, the input vector may consist of elements with a predefined order (e.g., these elements may indicate individual text characters in a text segment; or indicate data describing a continuous time environment). In the case where layer N is a hidden layer of a neural network, the activation value is the output from the previous (i.e., N-1) layer. The activation engine 222 is configured to use the input 205 and parameter matrix of layer N to generate multiple output values ​​225 using an activation function 228.

[0036] The layer N computing system 220 includes a continuous sparsity storage subsystem 224. The continuous sparsity storage subsystem 224 is a subsystem configured to obtain a parameter matrix for layer N. The parameter matrix consists of parameter values ​​stored in a continuous sparsity pattern, as described below. In one case, the sparsity pattern is such that it consists entirely of (i) zero values ​​and (ii) vertically extended blocks of non-zero values. In other words, in this case, the continuous sparsity storage subsystem 224 can obtain multiple continuous non-zero values ​​for a single input value. In some implementations, the continuous sparsity storage subsystem 224 implements an application programming interface (API) that takes as arguments multiple continuous parameter values ​​that are subsequently read from the parameter matrix for layer N.

[0037] The continuous sparsity storage subsystem 224 may be a separate storage subsystem for one or more layers, or the continuous sparsity storage subsystem 224 may be a global storage subsystem for the entire neural network system 210. In some implementations, the continuous sparsity storage subsystem 224 is implemented by a fast memory (e.g., SRAM) that is local to the processing component that performs the operation of the activation function 228. In other words, the parameter matrix of layer N may be stored in whole or in part in a memory local to the activation engine 222.

[0038] The layer N computing system 220 also includes a post-activation processing engine 226 that is configured to receive the output value 225 generated by the activation engine 222 and generate a layer N output 235 that is provided to the layer N+1 computing system 250 of the neural network system 210. The post-activation processing engine 226 can perform aggregation operations on the output 225 generated by the activation function 228, including summation, pooling, and soft-max operations.

[0039] To use the continuously stored parameter values, the activation engine 222 may read each non-zero value from the input 205 and then read multiple continuously stored parameter values ​​from the continuous sparsity storage subsystem 224. The activation engine 222 may then generate multiple output values ​​225 for a single input value, for example, by multiplying the input value by each of the continuously stored parameter values ​​and providing each product or partial sum of the products to the post-activation processing engine 226.

[0040] Figure 3 is a flowchart of an example process for training a neural network to have a parameter matrix having a continuous sparsity pattern. The example process can be performed by any suitable computing system that can implement operation and training updates of a neural network system. Optionally, Figure 3The process may be performed by a first computer system (e.g., a server) to generate a neural network model, and the neural network model may then be transmitted to a second computer system (e.g., a mobile phone or other client device) having relatively lower computing power and memory capacity. For convenience, the example process will be described as being performed by a system of one or more computers appropriately programmed according to the present specification.

[0041] The system performs training iterations (310). Generally, each training iteration updates the weights of a parameter matrix of the neural network after performing one or more inference passes through the neural network system. In some implementations, each training iteration performs a back-propagation process that updates the weights of the parameter matrix through the layers in a direction opposite to the direction of the inference pass.

[0042] The system determines whether to update the parameter sparsity pattern of the layer (320); in other words, reset the parameter values ​​so that the parameter matrix satisfies the continuous sparsity constraint. For example, the system can update the sparsity pattern after each training iteration, or periodically after a certain number of iterations, or only once after a predetermined number of iterations.

[0043] If the sparsity pattern is to be updated (320), the system can employ partitioning the parameter matrix into a plurality of nxm blocks, where n and m are different integers. The partitioning can be predefined (i.e., the positions of the blocks can be predefined) and can be the same each time the sparsity pattern is updated. The system sorts the partitioned blocks of parameters according to the absolute magnitude of the parameters in each block, for example, according to the sum of the absolute magnitudes of the parameters in each block (branching to 330); the system then clears all values ​​of the blocks whose parameters meet a threshold criterion indicating that the parameters are small. For example, the threshold criterion can be that the sum of the absolute magnitudes of the parameters of the block is less than a threshold (340). Clearing the value can include assigning a special value that will be ignored during inference, for example, zero or another reserved value (for simplicity, the term "zero value" is used here to cover both possibilities).

[0044] The threshold may be chosen so that the proportion of blocks whose values ​​are cleared is equal to a certain frequency value (which determines how many weights remain at non-zero values; in other words, the frequency value determines the sparsity of the resulting parameter matrix). The order of the blocks obtained in step 330 makes it easy to determine such a threshold. Note that in Figure 3 In a variation of the method, the threshold may be predetermined, and in this case the sorting step (330) may be omitted.

[0045] Step 340 has the effect of distilling the neural network, i.e. significantly reducing the amount of data required to store the neural network and the computing power required to operate the neural network. This makes the neural network suitable for use by a second computer system.

[0046] The system can be configured to use hyperparameters K, k, and Z, where K specifies the iteration interval (i.e., the number of iterations that occur between the times the system determines to update the parameter sparsity pattern in step 320), k specifies how many weights to retain at each update, and Z represents the maximum value of k. In some implementations, the value of k increases as training progresses according to the following formula:

[0047]

[0048] where t is the current iteration number, t0 is the iteration when the first update occurs, and S is the total number of update steps desired.

[0049] The system determines whether training is complete (350). If not, the system performs another training iteration (branch to 310).

[0050] If training is complete (350), the system optionally performs one or more pre-inference optimizations (360). The pre-inference optimizations are each designed to exploit continuous sparsity patterns in the newly trained parameter matrix. The pre-inference optimizations can include low-precision delta encoding, sparsity pattern unrolling, and precise load balancing.

[0051] First, the system can use low-precision delta encoding to encode the parameter matrix. Continuous sparsity allows additional data compression of the parameter matrix using delta encoding, which provides advantages over using the conventional compressed sparse row (CSR) format. In particular, the system can use a data type with lower precision than the data type encoding the parameters to delta encode the deltas between blocks. For example, instead of using 16-bit integers to encode the deltas, the system can use 8-bit integers instead. This reduces the storage space required to store a large amount of metadata about the parameter matrix, which makes the storage requirements of the entire model smaller.

[0052] Furthermore, the continuous pattern of parameter values ​​allows the system to unroll the sparsity pattern into the source code of the neural network kernel itself. In other words, the system can automatically generate source code for the neural network kernel that encodes the sparsity pattern in the code. This allows the system to skip the large number of memory accesses that would otherwise be needed to fetch data from memory that specifies the location of the next non-zero parameter value. Instead, the system can simply generate a source code statement that directly encodes the location of the next non-zero value. This also speeds up inference time by reducing branching and memory stalls.

[0053] The continuous pattern of parameter values ​​also means that the entire sparsity pattern can be encoded with fewer source code statements than a random sparsity pattern. This means that the entire neural network kernel can often fit entirely in a faster part of the memory hierarchy, such as the L1 cache instead of main memory. In some implementations, the entire kernel is small enough to fit in the L1 cache, while the encoded parameter matrix is ​​small enough to fit in the data cache.

[0054] Consider the following example 8x8 matrix with a continuous sparsity pattern exhibiting row sparsity:

[0055]

[0056] The compiler can unroll the program code that loops to read these parameter values, as shown in the pseudocode in the second column of Table 1. In other words, the source code does not have a for loop structure, but rather the source code has a series of sequential read instructions that read different parameter values ​​from the matrix. For clarity, other operations of the neural network kernel, including multiplication accumulation and other operations, have been omitted.

[0057]

[0058] Table 1

[0059] As can be seen from this example, unrolling generates only 64 read instructions to read all the values ​​of the parameter matrix.

[0060] The last column in Table 1 shows the expansion using the encoding. In particular, because the positions of consecutive non-zero values ​​of the parameter matrix are known at compile time, the system can generate read instructions that directly encode the positions of consecutive non-zero values.

[0061] In this example, the pseudocode instructions read a first variable that specifies the starting position of a block of non-zero parameter values, and a second variable that specifies how many consecutive values ​​to read. These are known in advance from the sorting and clearing process described above. Therefore, the system can read all non-zero values ​​from the example parameter matrix with only 7 read instructions instead of 64 read instructions.

[0062] This optimization also makes the code size smaller, which further makes the storage requirements of the model smaller.

[0063] Finally, the continuous sparsity pattern allows the system to perform very accurate load balancing across multiple compute element devices. Load balancing sparse matrix-vector multiplication operations with unknown sparsity patterns across multiple processing elements is challenging.

[0064] However, during inference, the model is fixed for a long period of time, and this allows the system to pre-compute the optimal load balancing among the N processors in advance. For example, the system can assign a substantially equal number of consecutive blocks or block parameter values ​​(i.e., parameter values ​​within one of the blocks) to each processing core of a CPU, GPU, or another parallel processing device, which enables faster computation of inferences than load balancing with a random sparsity pattern. As another example, the system can assign a substantially equal number of blocks to each processing element of a dedicated processor having multiple processing elements.

[0065] Figure 4 is a flow chart of an example process for performing an inference pass on a neural network having a parameter matrix having a continuous sparsity pattern. The example process may be performed by a computing system configured to implement operations of a layer of a neural network system. For example, Figure 2 The example processes may be performed by the Tier N computing system 220. For convenience, the example processes will be described as being performed by a system of one or more computers appropriately programmed according to the present specification.

[0066] The system receives an input vector (410). As described above, the input vector may be an output generated by a previous layer of the neural network system, or the input vector may be an initial representation of an initial input to the neural network system.

[0067] The system determines whether all input vectors have been processed (420).

[0068] If not, the system identifies the next non-zero input value of the input vector (branch to 430). In other words, the system identifies the next non-zero input value of the input vector that has not yet been processed. For devices with multiple processors, this may skip input values ​​processed by other processing components.

[0069] The system reads multiple consecutive non-zero parameter values ​​for the input value (440). The system can identify a row of the parameter matrix corresponding to the input value and read consecutive parameter values ​​from the row. If the system uses the above-described expansion using positional encoding, the system can immediately read the non-zero values ​​from the parameter matrix in constant time without searching.

[0070] The system generates multiple output values ​​for an input value and multiple consecutive non-zero parameter values ​​(450). For example, the system can multiply the input value by each of the multiple consecutive non-zero parameter values. The system can then optionally generate partial sums to be added to the results of other input values.

[0071] If all input values ​​have been processed (420), the process ends (branch to end).

[0072] Therefore, performing the inference pass using the continuous sparsity mode speeds up the matrix-vector multiplications involved in the inference pass.

[0073] Note that the above techniques can give advantages in parameter matrices with row sparsity or column sparsity. When the parameters have continuous column sparsity, the same input vector value can be read from memory once and used to multiply with multiple parameter values. This can be done before any further input vector values ​​are read from memory. When the parameters have continuous row sparsity, the input vector values ​​can be read with fewer memory operations. For example, a 1xn block size means that one memory operation can be used to load all one or more values ​​required for the block from the input vector, while an nx1 block size means that although multiple memory operations are required to load values ​​from the input vector, these values ​​can be reused n times. Blocks of size 1xn have the advantage that all memory loads of the input activations will be continuous, while blocks of size nx1 mean that multiple rows can share the same input activations.

[0074] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data device.

[0075] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or also include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0076] A computer program, which may also be referred to or described as a program, software, software application, APP, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and the computer program may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines or codes). A computer program may be deployed to run on one or more computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0077] For a system of one or more computers to be configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0078] The processes and logic flows described in this specification can be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit, such as an FPGA or ASIC, or by a combination of a special purpose logic circuit and one or more programmed computers.

[0079] The computer suitable for the operation of the computer program can be based on a general or special microprocessor or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Generally, the computer will also include one or more large-capacity storage devices (such as magnetic disks, magneto-optical disks or optical disks) for storing data, or operably coupled to one or more large-capacity storage devices to receive data from one or more large-capacity storage devices for storing data or to transmit data thereto, or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, just to name a few examples.

[0080] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0081] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse, trackball, or presence sensitive display or other surface through which the user can provide input to the computer). Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from a web browser on the user's device. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and receiving a response message from the user.

[0082] In addition to the above embodiments, the following embodiments are also innovative:

[0083] Embodiment 1 is a method comprising:

[0084] Receiving one or more parameter matrices of a neural network, wherein at least a first parameter matrix has parameter values ​​stored in a continuous sparsity pattern, the continuous sparsity pattern being a two-dimensional pattern consisting of a plurality of non-square rectangular blocks of zero values ​​and non-zero values, the number of zero values ​​being greater than the number of non-zero values;

[0085] storing the first parameter matrix in a storage device associated with the computing device;

[0086] receiving an input vector of activation values; and

[0087] Performing, by the computing device, an inference pass of the neural network to generate an output vector includes performing operations including:

[0088] reading one or more activation values ​​from the input vector from a storage device associated with the computing device,

[0089] reading a block of non-zero parameter values ​​from a storage device associated with the computing device, and

[0090] Multiply each of the one or more activation values ​​by one or more of the blocks of non-zero parameter values.

[0091] Embodiment 2 is a method according to embodiment 1, wherein the inference transfer comprises the following operations:

[0092] reading a first activation value from the input vector from a storage device associated with the computing device,

[0093] reading a plurality of consecutive non-zero parameter values ​​from a storage device associated with the computing device, and

[0094] A plurality of output values ​​of an output vector are generated by a process including multiplying a first activation value by a plurality of consecutive non-zero parameter values ​​before reading another activation value from the input vector.

[0095] Embodiment 3 is a method according to embodiment 1, wherein the inference transfer comprises the following operations:

[0096] reading a plurality of activation values ​​from the input vector from a storage device associated with the computing device,

[0097] reading a block of a plurality of non-zero parameter values ​​from a storage device associated with the computing device, and

[0098] An output value of an output vector is generated by a process including multiplying each of the plurality of non-zero activation values ​​by a corresponding one or more of the blocks of the plurality of non-zero parameter values ​​before reading another parameter value.

[0099] Embodiment 4 is the method of any of the preceding embodiments, further comprising:

[0100] receiving a value representing a number of processors of a computing device;

[0101] Counting the number of blocks of non-zero values ​​of the first parameter matrix; and

[0102] A substantially equal number of blocks or block parameter values ​​are distributed to each processor of the computing device.

[0103] Embodiment 5 is the method of embodiment 4, wherein the processor of the computing device is a processing core of a central processing unit (CPU).

[0104] Embodiment 6 is the method of embodiment 4, wherein the processor of the computing device is a processing element of a special purpose processor.

[0105] Embodiment 7 is the method of any one of embodiments 1-6, further comprising:

[0106] generating a plurality of source code statements encoding the positions of the next non-zero values ​​in the first parameter matrix,

[0107] Wherein performing an inference pass by the computing device includes executing the generated plurality of source code statements.

[0108] Embodiment 8 is the method of embodiment 7, wherein the source code statement omits one or more memory access instructions to read the location of the next non-zero value in the first parameter matrix.

[0109] Embodiment 9 is the method of any one of embodiments 1-8, further comprising:

[0110] Training a neural network involves performing operations including:

[0111] Upon determination that the magnitude of the one or more blocks of parameter values ​​satisfies the threshold criteria, zeros are assigned to the one or more blocks of parameter values.

[0112] Embodiment 10 is a method according to embodiment 9, comprising, prior to the step of allocating, a step of sorting the blocks of parameter values ​​according to a function of the parameter value of each block.

[0113] Embodiment 11 is the method of any one of embodiments 1-10, wherein each of the blocks has a size of nxm, wherein n and m are different integers, the step of reading the activation values ​​comprises reading m activation values, and the step of performing multiplication on the activation values ​​comprises multiplying each of the m activation values ​​by the corresponding n of the blocks of parameter values.

[0114] Embodiment 12 is the method of embodiment 11, wherein n is at least four times greater than m, or vice versa.

[0115] Embodiment 13 is the method of embodiment 11 or embodiment 12, wherein n is 1 or m is 1.

[0116] Embodiment 14 is the method of any one of embodiments 1-13, further comprising:

[0117] Deltas between non-zero parameter values ​​are encoded using a first data type having a lower precision than a second data type that encodes at least some of the non-zero parameter values.

[0118] Embodiment 15 is the method of any one of embodiments 1-14, wherein the neural network generates the original audio waveform in real time.

[0119] Embodiment 16 is the method of any of embodiments 1-15, wherein the computing device is a handheld computing device.

[0120] Embodiment 17 is the method of any of embodiments 1-16, wherein the computing device is mounted on an autonomous or semi-autonomous vehicle.

[0121] Embodiment 18 is a system comprising: one or more computers and one or more storage devices storing instructions, and when the instructions are executed by the one or more computers, the instructions are operable to cause the one or more computers to perform any of the methods of embodiments 1 to 15.

[0122] Embodiment 19 is a computer storage medium encoded with a computer program, the program including instructions, which, when executed by a data processing device, are operable to cause the data processing device to perform any one of the methods of embodiments 1 to 15.

[0123] Although this specification contains a number of specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope that may be claimed, but rather as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as acting in certain combinations, and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0124] Similarly, although the operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations should be performed in the particular order shown or sequentially, or that all of the operations shown should be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0125] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0126] In another example, although Figure 2 A feed-forward neural network is described where, in each inference pass, each neuron generates an output that is a function only of its input and corresponding network parameters, but in variations one or more of the neural layers may be recurrent neural layers where the output of a neural layer depends on the state of the neural layer in a previous inference pass. In one example, the layer N input 205 may include the layer N output 235 at one of a plurality of previous times, such as the layer N output 235 during a previous inference pass. In the case of a layer N computing system 220 constituting the first hidden layer, this may be added to the data input to the neural network. In the case of a layer N computing system 220 being a subsequent hidden layer of the neural network, this may be added to the output of the previous hidden layer.

Claims

1. A method comprising: Receiving one or more parameter matrices of a neural network, wherein at least a first parameter matrix has parameter values ​​stored in a continuous sparsity pattern, the continuous sparsity pattern being a two-dimensional pattern consisting of a plurality of non-square rectangular blocks of zero values ​​and non-zero values, wherein each of the rectangular blocks has a size nx1, n is greater than 1, and the number of zero values ​​is greater than the number of non-zero values; storing the first parameter matrix in a storage device associated with the computing device; receiving a value representing a number of processors of a computing device; Count the number of blocks with non-zero values ​​in the first parameter matrix; assigning a substantially equal number of blocks or block parameter values ​​to each processor of a computing device; receiving an input vector of activation values; and Performing, by the computing device, an inference pass of the neural network to generate an output vector includes performing operations including: reading a first activation value from an input vector from a storage device associated with the computing device, reading n consecutive non-zero parameter values ​​from a storage device associated with the computing device, and A plurality of output values ​​of the output vector are generated by a process comprising multiplying a first activation value by the n consecutive non-zero parameter values ​​before reading another activation value from the input vector.

2. The method according to claim 1, wherein: The processor of a computing device is the processing core of the central processing unit (CPU).

3. The method according to claim 1, wherein: A processor of a computing device is a processing element of a special-purpose processor.

4. The method according to claim 1, further comprising: generating a plurality of source code statements encoding the positions of the next non-zero values ​​in the first parameter matrix, Wherein performing the inference pass by the computing device includes running the generated plurality of source code statements.

5. The method according to claim 4, wherein: The source code statements omit one or more memory access instructions that read the location of the next non-zero value in the first parameter matrix.

6. The method according to claim 1, further comprising: Training a neural network involves performing the following operations: Upon determination that the magnitude of the one or more blocks of parameter values ​​satisfies the threshold criteria, zeros are assigned to the one or more blocks of parameter values.

7. A method according to claim 6, comprising, before the step of allocating, the step of sorting the blocks of parameter values ​​according to a function of the parameter value of each block.

8. The method according to claim 1, wherein: n is at least four.

9. The method according to claim 1, further comprising: Deltas between non-zero parameter values ​​are encoded using a first data type having a lower precision than a second data type that encodes at least some of the non-zero parameter values.

10. The method according to claim 1, wherein: The neural network generates the raw audio waveform in real time.

11. The method according to claim 1, wherein: The computing device is a handheld computing device.

12. The method according to claim 1, wherein: The computing device is installed on the autonomous or semi-autonomous vehicle.

13. A system comprising: One or more computers and one or more storage devices storing instructions, which, when executed by one or more computers, are operable to cause the one or more computers to perform the method of any one of claims 1 to 12.

14. A computer storage medium encoded with a computer program, the program comprising instructions which, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform the method of any one of claims 1 to 12.

Images