Methods, systems, and media for implementing a fast sparse neural network

Abstract

A neural network system is provided that includes at least one layer that applies a 1x1 convolution to a dense activation matrix using a kernel defined by a sparse weight matrix. The layer is implemented by a processor by accessing a sparsity dataset that indicates the locations of empty weights in the weight matrix. The processor selects feature values corresponding to other weights from a memory unit configured to store the activation matrix, and then uses these extracted feature values to compute the convolution values.

Description

Background Technology

[0001] This specification relates to neural network systems, particularly systems that can be implemented on processor devices with limited memory and / or processing power (such as mobile devices).

[0002] A neural network is a machine learning model that uses one or more units or nodes to predict the output of a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network—that is, the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters.

[0003] Many neural networks include convolutional layers, which have inputs and outputs defined based on arrays of elements. Typically, the array is a two-dimensional array, such as a rectangular array (e.g., a square array), with a width of W columns and a height of H rows. Each element is associated with an “input channel,” which is a feature vector consisting of C' feature values ​​(where C' is an integer, typically greater than 1). HW input channels are the inputs to the convolutional layer. Therefore, the set of input channels constitutes an activation matrix with C' × HW components. For each element, the convolutional layer multiplies the corresponding feature vector of the corresponding n × m portion of the array containing the element with the kernel, forming the corresponding “output channel” for that element. Each element’s output channel has C convolutional values ​​(where C is an integer, typically different from C'). A non-linear function (activation function), such as the rectified linear (ReLU) activation function, is typically applied to each of the C convolutional values ​​in the element’s output channel.

[0004] In an example called a "1x1 convolutional layer", n and m equal 1, so that the output channel of an element of the array depends on only the input channel of that element. In this case, the kernel applies a C'×C weight matrix ("weights") of the weight values ​​to the input channel of the element before applying the activation function (if any) to each convolutional value to generate the output channel of the element.

[0005] Convolutional Neural Networks (CNNs), which are neural networks that include at least one convolutional layer, have proven to be excellent at solving a wide variety of tasks. In many of these architectures, inference time, floating-point operations (FLOPs), and parameter counting are controlled by 1x1 convolutions, which are equivalent to matrix-matrix multiplications. Summary of the Invention

[0006] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations.

[0007] Generally, this disclosure proposes that a 1x1 convolutional layer applied to a dense activation matrix (i.e., where substantially all C' eigenvalues ​​of each element in the H×W elements are used in the process of generating the corresponding output channel of the element) uses a kernel defined by a sparse C'x C weight matrix. That is, more than half of the values ​​in the weight matrix are "empty" weights, i.e., weights that are not used when computing the convolutional values ​​and can be considered zero. By providing the processor with a sparse dataset (which is an indication of the location of empty weights in the weight matrix), the processor (only) selects eigenvalues ​​corresponding to other weights ("non-empty weights," defined as weights that are not empty weights) from memory cells configured to store the activation matrix, and then (only) uses these extracted eigenvalues ​​to compute the convolutional values. This can be done efficiently (e.g., in parallel or continuously over very short time intervals) for the corresponding convolutional values ​​of each of the many distinct elements in the array, because all elements require the same corresponding weights. That is, the corresponding convolutional values ​​of distinct elements depend on the same weights.

[0008] Multiple elements can reside in the same row of an array; that is, they can be "row vectors" of elements, for example, as a contiguous sequence of elements in that row. As is common in many known memory cells, memory cells used to store feature values ​​have a CHW-format memory layout, in which case selecting multiple elements as elements in the same row is particularly incentivized. In this case, the computation of the convolution vector can be efficiently performed using an inner loop of contiguous (typically non-overlapping) row vectors of elements in the same row and an outer loop of contiguous rows. Note that in implementations, the memory cell can be implemented as multiple physically separate memory devices.

[0009] The memory cell storing the activation matrix can be called the eigenvalue memory cell. The weight values ​​are stored in another memory cell called the weight memory cell. In the implementation, the weight memory cell can be the same memory device as the memory device implementing the eigenvalue memory cell or a different memory device.

[0010] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. First, the weight memory unit only needs to store non-empty weights, thus reducing the memory requirement of the weight memory unit compared to conventional neural networks with dense weight matrices of the same dimension. Second, the number of multiplication and / or division operations based on the activation matrix to generate output channels is reduced compared to conventional convolutional layers of neural networks, thereby improving energy efficiency and processing speed. Third, parallel processing hardware is effectively utilized because multiple convolution values ​​are computed in parallel. It is particularly efficient to have the memory unit store feature values ​​in a CHW memory layout if the processor is a GPU (Graphics Processing Unit) or other dedicated hardware. For example, CHW memory layers are particularly suitable for implementation in dedicated hardware that performs matrix multiplication in hardware (e.g., a Tensor Processing Unit (TPU) or another hardware machine learning accelerator). Experiments have shown that all these advantages can be achieved with minimal performance degradation compared to conventional neural networks, for example, in the proposed neural network, the proportion of empty weights (sparseness) is no greater than 95%.

[0011] For example, for some known test problems, the inventors have experimentally discovered surprising results: by selecting at least 70% and / or no more than 95% of the weights in at least one convolutional layer of a neural network trained to solve a standard test problem as empty (i.e., sparsity within the "sparseness range" of 70% to 95%), the number of parameters of the neural network is reduced by more than a factor of 5, and the number of computations (floating-point operations—FLOPS) performed by the neural network to generate output from input is reduced by up to a factor of 3, while the performance degradation of the neural network is less than 1%. It was also found that when the neural network is implemented on low-power processors (such as the CPUs of mobile devices such as mobile phones, tablets, or other forms of mobile client devices), the processing time of the neural network is reduced by 1.3 to 2 times. This makes the current neural network particularly suitable for such devices. While sparsity in neural networks is not a new concept, it is often considered a practical method for accelerating models because the common misconception is that sparse operations cannot be fast enough to achieve a real speedup during perturbations. The current experiments show that this is not the case. One reason is that in these experiments, the sparse weight matrix is ​​combined with the dense activation matrix to provide sparsity. Similarly, while previous work typically focused on extremely sparse problems (sparseness exceeding 99%), the sparsity range used in this experiment was relatively low.

[0012] Each non-empty weight can take any value (e.g., a floating-point value or any integer value within a predefined range), and in some implementations, even optionally zero; that is, in these implementations, some non-empty weights may be equal to zero, even though the neural network implementation does not take into account prior information that they are zero. Typically, weights are generated during training, and in principle, the training process may generate some weights equal to zero without marking them as empty weights, and therefore they are not included in the empty weight indication. Alternatively, in other implementations, all weights that are zero after training (or are set to zero because they conform to predetermined criteria after training, such as having a magnitude below a predetermined threshold after training) are marked as empty weights and included in the empty weight indication; in other words, in these implementations, all non-empty weights are non-zero. In other implementations, empty weights can be specified before training so that the training process only modifies the values ​​of the non-empty weights.

[0013] Note that although this disclosure proposes a sparse weight matrix, there is essentially no sparsity in the activation matrix (the input to the convolutional layer). Therefore, the convolutional layers of the neural network are configured to sequentially select consecutive row vectors of elements from the array (e.g., each element has the same number of elements), and each row vector is processed in parallel as described above, so that the convolutional layers of the neural network ultimately process multiple consecutive row vectors that collectively comprise all the elements of the array. There may be essentially no "empty" eigenvalues ​​(i.e., the eigenvalues ​​that the convolutional values ​​depend on).

[0014] Not all layers of a neural network need to use the principles described above. For example, the input layer of a neural network can be fully connected, and / or the output layer can be fully connected. In other words, sparsity can be provided at specific locations within the neural network, i.e., locations stimulated by the capabilities of a known computer processor capable of parallel processing.

[0015] Optionally, the weight matrix can have a certain regularity. For example, for multiple rows of the weight matrix (corresponding to multiple convolution values ​​for each element), the non-empty weights can be in the same position (i.e., corresponding to the same subset of eigenvalues ​​for the input channels of each element). Therefore, different convolution values ​​among the convolution values ​​of a given element depend on the same subset of eigenvalues ​​for the input channels of the element. In this case, computational efficiency can be further improved by computing those convolution values ​​for the output channels of the row vectors in parallel (as “blocks”). Furthermore, for a given number of non-empty weights, the size of the sparse dataset can be reduced due to the regularity of the arrangement of the non-empty weights.

[0016] To express this differently, the rows of the weight matrix can be vertically divided into groups of weight rows (“weight rows”), where each group can be an equal number of weight rows, at least two rows, and can consist of consecutive weight rows. For each group, the non-empty weights of all weight rows in that group can be located at the same position along the weight rows (i.e., corresponding to the same subset of eigenvalues). In this case, when processing the row vectors of the elements to generate the corresponding convolution values, the groups of weight rows can be processed sequentially, but the weight rows of each group can be processed in parallel to generate the corresponding convolution values.

[0017] The values ​​of H and W can be any integers, but typically each value is at least 10. The number of eigenvalues ​​for each input channel C' is typically at least 2, and can be greater than 2. The number of elements in each row vector is at least 2, more typically at least 3 (such as 4 or 8) or at least 8 (such as 16).

[0018] Preferably, during the generation of each convolutional value for multiple elements, once the feature values ​​of these multiple elements (whose corresponding weights are non-empty) are extracted from the memory cells (e.g., simultaneously), the extracted feature values ​​are stored (e.g., simultaneously) in a cache memory. However, for feature values ​​that were stored in the cache memory when generating the preceding convolutional values ​​for the multiple elements, extraction and storage are preferably not performed. This further saves computation time.

[0019] In addition to retrieving the feature values ​​of multiple elements (for which convolution values ​​are currently being computed) from memory cells and writing them to cache memory, the corresponding feature values ​​of multiple additional elements are also read from memory cells (e.g., simultaneously) and written to cache memory (e.g., simultaneously). The convolution values ​​of these additional elements are not generated in parallel with the convolution values ​​of the multiple elements; for example, this might be done in a separate processing loop. However, the fact that they have already been pre-fetched means it's time to compute convolution values ​​for these elements, as their feature values ​​already exist in cache memory. This reduces the computation time required to generate convolution values ​​for these additional elements.

[0020] The concepts presented in this paper can be represented as methods for implementing neural networks or, alternatively, as methods for training neural networks. This training process typically involves deriving the weight matrix.

[0021] Training may involve designating at least some weights of the weight matrix as empty weights (i.e., selecting some components of the weight matrix as empty weights). This can be done by performing a training process to obtain weights, identifying result weights that meet certain criteria (e.g., having amplitudes below a threshold), and designating the identified weights as empty weights. More sophisticated criteria for identifying result weights to designate as empty weights may consider multiple result weights simultaneously, for example, ensuring that the identified result weights are weights with relatively low amplitudes, constrained by the regularity of the position of the empty weights in the weight matrix. Alternatively, some or all of the empty weights may be pre-designated (i.e., a sparse dataset may exist before training the non-empty weights) to ensure regularity in the arrangement of the non-empty weights discussed above.

[0022] Alternative expressions of this concept may be based on a computer system comprising: one or more computers located at one or more locations and configured to perform a method; or a computer program product comprising instructions, such as a non-transitory computer storage medium (memory device) storing program instructions or downloadable software comprising instructions, wherein the instructions, when executed by one or more computers, cause the computers to perform any of the methods described above.

[0023] There are many applications for implementing neural networks. In a broad sense, the system can be used in any neural network that receives input data with dimensions corresponding to the dimensions of the array (e.g., two-dimensional).

[0024] For example, the input data can represent a still or moving image, in which case the data values ​​can represent pixel values. The input data can be real-world data, such as data collected by one or more sensor devices (such as one or more still and / or video cameras).

[0025] For example, a neural network can be used as a classifier that is trained to classify input data into one or more classes. For instance, a neural network system can be used to classify an image (e.g., from a real-world or simulated environment) into one of a predetermined number of classes.

[0026] Alternatively, for example, a neural network can be used as a generative model, for instance, to generate examples conditioned on edge information. Alternatively, it can be used to score the quality of already generated examples, i.e., based on how well the examples match the training data.

[0027] Alternatively, neural networks can be used for reinforcement learning, for example, to generate control data to control agents (e.g., robots) moving in real-world or simulated environments. Alternatively, neural network systems can be trained to generate data that predicts future image or video sequences seen by real or virtual cameras associated with physical objects or agents in simulated or real-world environments. Attached Figure Description

[0028] For the sake of illustration only, an example of this disclosure will now be described with reference to the following figure, in which:

[0029] Figure 1 A neural network employing the currently disclosed method is shown.

[0030] Figure 2 The following is shown for implementation Figure 1 Computer systems with neural networks.

[0031] Figure 3 The diagram illustrates the process of... Figure 1 The first convolution operation is performed by the layers of the neural network.

[0032] Figure 4 The diagram illustrates the process of... Figure 1 The layers of the neural network perform a second alternative convolution operation.

[0033] Depend on Figures 5(a) to 5(e) Figure 5 schematically illustrates the process of execution. Figure 3 A series of memory operations performed during the convolution operation.

[0034] Figure 6 This illustrates the process during which the following occurs (e.g., the process in Figure 5) Figure 1 The steps of the method for executing a neural network.

[0035] In different drawings, the same reference numbers and titles all indicate the same element. Detailed Implementation

[0036] Figure 1 A neural network 100 is shown as an example of this disclosure. The neural network 100 may be implemented by one or more computer systems located at one or more locations.

[0037] Neural network 100 includes an input layer 101, an output layer 103, and one or more hidden layers 102a, 102b, 102c. The input layer 101, hidden layers 102a, 102b, 102c, and output layer 103 are arranged sequentially. The output of each layer except output layer 103 provides input to the next layer in the sequence. One or more of the input layer 101, hidden layers 102a, 102b, 102c, and output layer 103 are convolutional layers. In practice, they may all be convolutional layers, although at least output layer 103 is typically not. Each convolutional layer receives input defined as an array of elements (typically a two-dimensional array). For each element, there is a corresponding input channel, which is an eigenvector consisting of C' eigenvalues. Similarly, for each element, the convolutional layer generates a corresponding output channel, which has a C value called a "convolution value". Each convolutional layer employs a corresponding kernel defined by a weight matrix.

[0038] The input to input layer 101 is data that defines the image, such as data that assigns values ​​for one or more values ​​to each pixel in an array of pixels. Pixels can correspond to a corresponding element. For example, for this layer, C' can be 3, and the feature values ​​for the input channels of each element can be the intensities of the red, green, and blue channels, respectively.

[0039] At least one of the layers, particularly one of the hidden layers 102a, 102b, and 102c, is a 1x1 convolutional layer. In the case of a 1x1 convolutional layer, the output channel of each element depends only on the input channel of the element. That is, the kernel does not contain weights that cause the component of one element's output channel to depend on the input channel of another element.

[0040] As described below, one or more layers of the neural network 100 implementing 1x1 convolutions can be implemented using kernels that exhibit “sparseness” (i.e., at least a certain percentage of the weights are zero, for example, at least half), particularly one of the hidden layers 102a, 102b, and 102c. However, not all layers of the neural network exhibit sparsity.

[0041] First, the input layer 101 may include kernels that do not exhibit sparsity. Their contribution to parameter counting, FLOP counting, and runtime is minimal. Instead, the input layer 101 may employ dense convolutional kernels and take an image as its input.

[0042] Similarly, one or more of layers 101, 102a, 102b, 102c, and 103 can implement “squeeze and excitation” (SE) layers, as described in “Squeeze and excitation networks” (2019) by Jie Hu et al. In such layers, the input to the layer is mapped to a feature map denoted by U (e.g., via convolution), and the feature map undergoes a “squeeze” operation, which aggregates the feature maps across their H×W spatial dimensions to produce channel descriptors, resulting in embeddings of the globally distributed channel-based feature responses. This aggregation is followed by an “excitation” operation, which takes the embeddings as input and produces a set of weights for each channel, which are applied to the feature map U to generate the output of the SE layer. If such an SE layer exists in neural network 100, it may not employ the sparse kernels described below, as experiments have shown that they typically contribute less than 1% to the total FLOPs of dense models that routinely use them.

[0043] Similarly, the last layer 103 of neural network 100 can be implemented as a fully connected layer instead of a convolutional layer. Again, experiments show that in traditional models, the fully connected output layer contributes negligible (<1%) to the total FLOP count, but contributes a significant portion (20% to 50%) of the total parameters, especially when the training of the neural network prunes other layers.

[0044] Figure 2 The diagram illustrates the implementation. Figure 1 A computer system 200 provides a neural network 100. The computer system 200 receives data input 201, which may be image data describing one or more images. The computer system 200 includes a processor 202 and memory units 203, 204, and 205. The processor 202 may be a processor capable of processing multiple computation threads simultaneously. The first memory unit among memory units 203, 204, and 205 is a program memory unit 203, which stores program instructions that operate to control the processor 202 to implement the neural network, specifically performing convolution operations on hidden layers 102a, 102b, and 102c as described below. The second memory unit among the memory units is a weight memory unit 204, which stores weights that define the operations performed by the layers of the neural network 100. For each layer, there exists a corresponding weight matrix composed of weights. For each layer, the weight memory unit 204 may also store a corresponding sparse dataset, which indicates one or more non-empty weight values ​​of the corresponding weight matrix for each output channel.

[0045] The third memory unit in the memory cell is feature value memory unit 205, which stores data input to and output from each layer. Upon receiving data input 201, the data is stored in feature value memory unit 205.

[0046] The data in data input 201 and stored in feature value memory 205 can be in a standard HWC layout, where the values ​​of different channels corresponding to a spatial location are adjacent in memory. That is, if the number of elements in each row of the array is represented as W, the number of rows in the array is represented as H, and the number of channels in each element is represented as C, the memory location of the c-th channel value of the element at position (h, w) in the array (i.e., the offset distance from any arbitrary location in the memory space) can be represented as h*(W)*(C)+w*(C)+c. After receiving data input 201, data input 201 can generally still be stored in feature memory unit 205 in HWC format.

[0047] To implement a layer of neural network 100, processor 202 can transfer consecutive portions of the input data describing the layer from feature memory unit 205 to cache memory 206 of processor 202. In the case of layers exhibiting sparsity, for each element, the transfer can be performed in multiple steps, in each step, only a subset of the feature values ​​of the input channels of that element is transferred to cache memory 206 as needed to generate portions of the output channels of the element. To allow the convolution values ​​of multiple elements to be generated together (e.g., in parallel), the feature values ​​of multiple elements can be transferred from feature memory unit 205 to cache memory 206 simultaneously.

[0048] For each layer (optionally, except for output layer 103), the convolution value of the corresponding output channel of each element is stored in feature value memory unit 205. The output channels are then read from feature value memory unit 205 by processor 202 and used by processor 202 as input data for successive layers of neural network 100. As described below, the output channels of one or more layers of neural network 100 (such as input layer 101 and / or one or more hidden layers 102a, 102b, 102c) can be stored in feature value memory unit 205 in CHW format (also referred to herein as CHW layout). In CHW layout, the values ​​of all spatial locations of a channel are adjacent in memory. In CHW layout, the memory location (offset from any position in memory space) of the c-th channel of the element at position (h, w) in the H×W array is c*(W)*(H) + h*(W) + w. If the input data of one or more of the hidden layers 102a, 102b, 102c and the output layer 103 is in CHW format, especially the input data of the convolutional layer 102a immediately following the input layer 101 is in CHW format, then sparse convolution operations are convenient.

[0049] The output channel of output layer 103 is transmitted from computer system 200 as output data 207. For example, the output may represent a classification of image data 201. Alternatively, if data input 201 is side data and neural network 100 is a generative network, the output data 207 may be a dataset representing an image or signal (such as a sound waveform). Alternatively, if data input 201 is sensor data describing an environment, such as images of a real-world environment collected by a still or video camera, the output data 207 may be control data transmitted to an agent to control the agent's interaction with the environment, such as moving within the environment (by translation, rotation, and / or reconfiguration). Alternatively, if data input 201 is data representing a portion of natural language (e.g., a sequence of letters or sound signals collected by sensors when natural language is spoken), the output data 207 may be modified natural language, such as a translation of natural language, and may also be a sequence of letters or sound signals.

[0050] Turn Figure 3 A schematic diagram is shown, explaining the execution of a 1x1 convolution operation through one of the layers of a neural network 100, such as hidden layers 102a, 102b, and 102c, using the sparsity principle disclosed herein. The input to the convolution operation is an activation matrix 301 in CHW format. Each column of the activation matrix 301 represents an input channel to one of the elements of an array consisting of C' eigenvalues. The corresponding eigenvalues ​​of the activation matrix 301 are... Figure 1 The corresponding box in one of the columns is used to represent this. Figure 3In this context, the activation matrix 301 is represented by the number of columns with a height × width (i.e., HW) and the number of rows represented as "channels in (input channels)" C'. In a sense, the activation matrix 301 is dense, meaning that virtually no eigenvalue for any channel of any element is "empty," i.e., it is known in advance that it is zero (e.g., it is known in advance that there are no values ​​or that no more than 1% of the values ​​are zero). In fact, all or virtually all C' × HW values ​​are potentially non-zero. Non-empty eigenvalues ​​in... Figure 3 The boxes in the activation matrix 301 are represented by shaded boxes; that is, all boxes in the activation matrix 301 are shaded.

[0051] The kernel of a 1x1 convolutional layer is represented by a CxC' weight matrix 302, where C is the number of convolutional values ​​in the output channel of each element. C can be the same as or different from C'. Zero values ​​("null values") in the weight matrix 302 are represented by unshaded (white) boxes, while non-zero values ​​("non-null values") in the kernel matrix are represented by shaded boxes. The proportion of non-null values ​​is small, for example, in the range of 25% to 10%. The convolution operation consists of multiplying the weight matrix 302 by the activation matrix 301. This is described below with reference to FIG5.

[0052] Figure 4 The illustration shows an alternative form of the 1x1 convolution operation that can be performed by one of the hidden layers 102a, 102b, and 102c. Activation matrix 401 and... Figure 3 Same as in, but with Figure 3 The difference is that, in Figure 4 In this case, the rows of weight matrix 402 (“weight rows”) are vertically divided into groups 403, 404, 405, and 406. For each group, the non-empty weights of all weight rows in that group are located at the same position along the weight rows (i.e., corresponding to the same subset of eigenvalues). Each group can consist of an equal number of weight rows, which is at least 2 (in the case of weight rows). Figure 4 In the text, each group (403, 404, 405, 406) has 4 rows. For example... Figure 4 As shown, the weight rows of each group 403, 404, 405, and 406 are consecutive weight rows, but in an alternative arrangement, the rows of the group can be interleaved.

[0053] When processing each column of matrix 401 (i.e., the input value of each element) to generate the corresponding convolutional value, the weight rows of each group can be processed in parallel to generate the corresponding convolutional value. However, the weight rows of different groups can be processed sequentially.

[0054] Figure 5 shows the evaluation. Figure 3The memory read / write operations of the kernel in one of the 1x1 convolution operations. For simplicity, the number of rows C' of the activation matrix (i.e., the number of input channels per element) is 4 in this example, and the number of output channels C per element is also 4. However, the scheme in Figure 5 can be easily extended to values ​​C' and C being any positive integers (e.g., ...). Figure 3 and Figure 4 The cases where C' = 8 and C = 16 are equal to or different from each other. Specifically, Figure 5(a) shows an example weight matrix 501 with non-zero (non-empty) elements represented by boxes containing intersections and zero (empty) elements represented by boxes without intersections. For example, the fourth output channel (fourth row) has non-empty weight values ​​only for the second and fourth input channels.

[0055] Figures 5(b) to 5(e) The operation sequence is shown, in which the values ​​of all four channels of the eight elements of the array are processed together, but the number of elements processed together may vary. For example, 16 elements may be processed together, and these elements may correspond to a cache line in cache memory 206. The memory space of the cache memory, denoted by 502, has a number of rows (cache lines) equal to (at least) the number of feature values ​​in each input channel. In each row, there are multiple memory locations, each memory location being configured to store the corresponding feature value. Figures 5(b) to 5(e) The memory space 503 of the feature value memory unit 205 is also shown, which is used to store the convolution values ​​as the output of the 1x1 convolutional layer.

[0056] In the first step shown in Figure 5(b), processor 202 determines the positions of non-zero weights in the first row of the weight matrix from the sparse dataset. In this example, the first row of weight matrix 501 has non-null values ​​at the first and fourth positions. For a set of eight elements of the array, in the corresponding rows of memory space 502 corresponding to the non-zero weights in the first row of weight matrix 501, processor 202 reads the eigenvalues ​​corresponding to these non-zero weights from eigenvalue memory unit 205 and writes them to the first eight positions of memory space 502 in cache memory 206. That is, the first eigenvalue of the set of eight elements is written to the first eight positions 5021 of the first row of memory space 502, and the fourth eigenvalue of the set of eight elements is written to the first eight positions 5022 of the fourth row of memory space 502. The eigenvalues ​​written to positions 5021 and 5022 are represented by cross boxes. These read and write operations are essentially performed simultaneously on the eight elements (spatial positions in the array).

[0057] Optionally, for each non-empty weight value in the first row of weight matrix 501 (i.e., the first weight and the fourth weight), the processor also reads, for example, the corresponding eigenvalues ​​of the second set of 8 elements (i.e., the first eigenvalue and the fourth eigenvalue) and writes them to the next 8 positions 5023, 5024 (i.e., the first and fourth rows) of the corresponding row in memory space 502. They are shown as boxes in Figure 5(b), with a diagonal line running from the top left to the bottom right through them. These pre-fetched eigenvalues ​​are later used (after all the convolution values ​​of the first set of 8 elements have been generated) to generate the convolution values ​​of the second set of elements.

[0058] For each of the first set of eight elements, processor 502 forms the corresponding convolutional value by multiplying each non-empty weight in the first row of weight matrix 501 with the eigenvalue of the element corresponding to the non-empty weight in the row of memory space 502 and accumulating (adding) the results. Processor 202 then writes the corresponding convolutional value of each of these eight elements into the first eight locations 5031 of the memory space portion 503 of eigenvalue memory unit 205. Optionally, a nonlinear function (e.g., ReLU function) included in the 1x1 convolution can be applied to each convolutional value. Thus, the process shown in FIG5(b) has generated the first convolutional value for the output channel of the first set of eight elements. The convolutional values ​​of the first set of eight elements can be generated in parallel (or sequentially in short time intervals) and can be written to eigenvalue memory unit 205 substantially simultaneously.

[0059] As described above, processor 202 may have optionally written the first and fourth eigenvalues ​​of the second set of eight elements to eight corresponding memory locations 5023, 5024. In this case, processor 202 may optionally generate convolution values ​​for the second set of eight elements through the same process. That is, for each of the eight elements in the second set, processor 202 accumulates (adds) the results by multiplying each non-empty weight in the first row of weight matrix 501 with the eigenvalue of that element corresponding to the non-empty weight in the portions 5023, 5024 of the row of memory space 502, and writes them to the next eight locations 5032 of the first row of memory space 503 to form the corresponding convolution value. This is performed for each convolution value if the 1x1 convolution operation involves performing a non-linear function. Note that this process is not illustrated in Figure 5(a) because the process of calculating the corresponding convolution value for the first output channel of each of the eight elements in the second set may optionally be performed after completing the calculation. Figures 5(b) to 5(e) The steps shown in the diagram are executed after the sequence of steps.

[0060] Figure 5(c) illustrates how the same process shown in Figure 5(b) is performed to compute the second convolution value of the output channel of the first set of 8 elements. In this example, the non-empty weight values ​​are at the second and third positions of the second row of the weight matrix 501. Therefore, the processor reads the second and third eigenvalues ​​of the input channel of the first set of 8 elements from the eigenvalue memory unit 205 and writes them to the positions in the memory space 502 indicated by the dots. In this example, the non-empty weights in the second row of the weight matrix 501 occur at different positions than the non-empty weights in the first row of the weight matrix 501. However, if any non-empty weights are at the same positions (i.e., associated with the same input channel), the read / write operation for that input channel can be omitted because the memory space 502 already contains these eigenvalues.

[0061] Optionally, the second and third eigenvalues ​​of the second set of 8 elements are written to the next 8 positions (as shown by a diagonal line from the bottom left to the top right) of the corresponding rows (i.e., the second and third rows) of the memory space 502. Then, the processor 202 calculates the corresponding second convolution value for each of the output channels in the first set of 8 elements by multiplying the non-empty weights in the second row of the weight matrix 501 with the corresponding eigenvalues ​​of the first set of 8 elements and adding the results.

[0062] Figures 5(d) and 5(e) illustrate how the processor generates convolution values ​​for the third and fourth convolution values ​​of the output channels of the first set of eight elements, respectively. Note that after the processes shown in Figures 5(b) and 5(c), all eigenvalues ​​of the first set of eight elements (spatial locations) are in cache memory 206; therefore, processor 202 can generate convolution values ​​for the remaining output channels without reading any more data from eigenvalue memory unit 205. To generate the third and fourth convolution values ​​for the output channels of each of the first set of eight elements, processor 202 multiplies the non-empty weights in the third and fourth rows of weight matrix 501 with the corresponding eigenvalues ​​of the first set of eight elements, respectively, and appends the results. This means that loading eigenvalues ​​to perform the multiplications in steps 5(d) and 5(e) is fast, even though eigenvalue memory unit 205 and cache memory 206 are accessed randomly.

[0063] exist Figures 5(b) to 5(e) In the step sequence shown, the outer loop is above the columns, and the inner loop is above the rows. This allows each stripe at each of the 16 spatial locations in the activation matrix to remain in the cache memory 206 until it is no longer needed. The steps in Figures 5(b) and 5(c) prepare the cache memory 206, while subsequent steps in Figures 5(d) and 5(e) load all eigenvalues ​​from the cache memory 206.

[0064] If any regularity, even a small one, is known in the structure of the weight matrix 501, this allows the process in Figure 5 to vary significantly in performance by increasing data reuse after loading the weights and eigenvalues ​​into the registers of the processor 202. For example, as mentioned above... Figure 4 As described, multiple output channels of the weight matrix can have the same zero / non-zero weight pattern. Alternatively or additionally, multiple columns of the weight matrix can have the same zero / non-zero weight pattern. The training process for generating the weight matrix is ​​constrained to produce sparse patterns so that multiple output or input channels share the same zero / non-zero pattern, creating 'blocks' in the weight matrix, such as... Figure 4 As shown, a block is a set of multiple rows with the same sparse pattern. Figure 4 As shown, creating blocks in the output channel dimension allows for more data reuse compared to forming blocks in the input channel dimension. Experiments conducted indicate that either choice has the same impact on accuracy, but setting multiple rows of the weight matrix to have the same pattern (e.g., ...) Figure 4 As shown in Figure 5, this leads to higher processing efficiency, and is therefore the preferred method. In some experiments, the constraint for training the weight matrix is ​​that the groups consist of 2 rows or 4 rows. In this case, the inner loop can include generating output channels for all rows of the corresponding group. For example, in the case where each group contains 2 weight rows, a single inner loop performed on the feature vectors of the array (e.g., 8 elements per row of the array) can generate two corresponding convolution values ​​for the output channels of the set of elements. In practice, the scheme in Figure 5 is varied so that... Figures 5(a) to 5(b) Each stage in the process is replaced with a stage in which all weight rows of one of the groups are used to generate all the corresponding convolution values.

[0065] exist Figure 6 The method 600 for generating convolutional values ​​in the process shown in Figure 5 is summarized below. In step 601, for each convolutional value, one or more feature values ​​are obtained from the activation matrix based on the sparse dataset (which acts as an indicator of the non-empty weights corresponding to each convolutional value in the weight matrix). In step 602, the convolutional value is generated as the sum of the corresponding extracted feature values ​​weighted by the corresponding non-empty weights.

[0066] The experiments conducted demonstrate that using the above techniques can significantly reduce computational burden and memory requirements. Three factors particularly contribute to this:

[0067] 1. Although the weight matrix is ​​sparse, the activation matrix is ​​dense. This means that the processor 202 can execute vector loads from the activation matrix and process multiple spatial locations simultaneously.

[0068] 2. By processing the matrix in the correct order, the system can retain values ​​that will be randomly accessed in the cache memory. Note that random access from cache memory 206 can be faster than random access from eigenvalue memory unit 205.

[0069] 3. In particular, when the number of input channels is small, the number of eigenvalues ​​prefetched from the activation matrix of the second set of elements is further reduced in cases where the cache memory 206 does not contain the required eigenvalues ​​when the convolution value of the second set of elements is to be calculated, so that the value must be obtained from the eigenvalue memory unit 205.

[0070] Experiments show that, for a constant computational budget, sparse convolutional networks are more accurate than dense convolutional networks. For example, in clockwork time, their accuracy is 1.3 to 2.4 times that of dense convolutional networks, requiring only approximately 66% of the parameters—roughly equivalent to an entire generation of improvement.

[0071] This specification uses the term "configured to" in relation to system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform the operation or action during operation. For one or more computer programs to be configured to perform a particular operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0072] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated for encoding information to be transmitted to a suitable receiver device for execution by the data processing apparatus.

[0073] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include a dedicated logic circuit system, such as a FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0074] A computer program (which may also be referred to or described as a program, software, software application, application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinating files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on a single computer or on multiple computers located on a single station or distributed across multiple stations and interconnected by a data communication network.

[0075] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way or is completely unstructured, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed in different ways.

[0076] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed on and run on the same one or more computers.

[0077] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a special-purpose logic circuit system (e.g., an FPGA or ASIC) or a combination of a special-purpose logic circuit system and one or more programmable computers.

[0078] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to or both of these mass storage devices. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0079] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROMs, and DVD-ROMs.

[0080] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form (including sound input, voice input, or tactile input). Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending web pages to a web browser on the user's device in response to a request received from a web browser. Similarly, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user as an answer.

[0081] For example, the data processing apparatus used to implement machine learning models may also include dedicated hardware accelerator units for processing the general and computationally intensive parts of machine learning training or production, namely, inference and workloads.

[0082] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0083] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or application through which a user can interact with the implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0084] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device (e.g., to display data to a user interacting with the device (which acts as a client) and to receive user input from that user). Data generated at the user device (e.g., the result of user interaction) may be received from the device at the server.

[0085] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described within the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations, and initially even claimed in the same manner, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0086] Similarly, although the operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential order, or that all the operations shown be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0087] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions referenced in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily need to be performed in the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for implementing a neural network, the neural network comprising a plurality of layers, the plurality of layers including at least one sparse 1x1 convolutional layer, the input of the sparse 1x1 convolutional layer comprising a corresponding input channel of a feature value for each of a plurality of elements set in an H×W array. The sparse 1x1 convolutional layer is configured to apply sparse 1x1 convolutions to the input channels to form corresponding output channels. Each output channel consists of multiple convolutional values. The sparse 1x1 convolutions are defined by a sparse C×C' weight matrix, which has multiple empty weights equal to zero and multiple non-empty weights. The input channels form a dense C'×HW activation matrix, which has eigenvalues ​​defined for each element of the dense C'×HW activation matrix. The method includes: Obtain an indication of the empty weights of the sparse C×C' weight matrix stored in the first memory unit; as well as The sparse C×C' weight matrix is ​​processed by combining the dense C'×HW activation matrix with the following steps: Convolution values ​​of the multiple elements in the row vectors containing the multiple elements of the dense C'×HW activation matrix are generated from the elements of the row vectors. (a) Extract the corresponding feature values ​​of the input channel from the second memory cell storing the dense C'×HW activation matrix, wherein the corresponding feature values ​​are non-empty weights of the sparse C×C' weight matrix for these feature values, according to the instruction. as well as (b) Form a corresponding sum of the extracted corresponding feature values ​​weighted by the corresponding non-empty weights.

2. The method according to claim 1, wherein, The empty weights constitute 70% to 95% of the components of the sparse C×C' weight matrix.

3. The method according to claim 1, wherein, The output layer of the neural network is fully connected.

4. The method according to claim 1, wherein, The second memory cell has a CHW memory layout.

5. The method according to claim 4, wherein, The processing is performed using an inner loop of consecutive row vectors of elements in the same row and an outer loop of consecutive rows.

6. The method according to claim 1, wherein, The process is performed repeatedly on consecutive row vectors, which together comprise an entire array of elements.

7. The method according to claim 1, wherein, The neural network further includes an output layer following the sparse 1x1 convolutional layer and configured to generate one or more output values, each output value being determined based on all convolutional values ​​of all said elements.

8. The method according to claim 1, wherein, The non-empty weights are at the same position in each of the multiple rows of the sparse C×C' weight matrix.

9. The method according to claim 8, wherein, The processing of multiple rows of the sparse C×C' weight matrix is ​​performed in parallel to generate multiple convolution values ​​corresponding to the output channels of the row vectors.

10. The method according to any one of claims 1 to 9, wherein, During the generation of the convolution values ​​of the plurality of elements, after the corresponding feature values ​​are extracted from the second memory unit, the extracted feature values ​​are stored in a cache memory. During the generation of the previous convolution values ​​of the plurality of elements, the feature values ​​stored in the cache memory were not extracted and stored.

11. The method according to claim 10, wherein, During the generation of the convolution values ​​of multiple elements based on the corresponding feature values ​​of the multiple elements, the corresponding feature values ​​of multiple additional elements are also read from the second memory unit into the cache memory, and the convolution values ​​of the multiple additional elements are not generated in parallel with the convolution values ​​of the multiple elements.

12. A method for generating a neural network according to any one of claims 1 to 11, the method comprising: The training process is performed based on training examples to determine the non-empty weights of the sparse C×C' weight matrix.

13. The method according to claim 12, wherein, The null weights of the sparse C×C' weight matrix are predetermined based on the sparse dataset.

14. A system configured to implement a neural network, the neural network comprising a plurality of layers, the plurality of layers including at least one sparse 1x1 convolutional layer, the input of the sparse 1x1 convolutional layer comprising a corresponding input channel of a feature value for each of a plurality of elements set in an H×W array. The sparse 1x1 convolutional layer is configured to apply sparse 1x1 convolutions to the input channels to form corresponding output channels. Each output channel consists of multiple convolutional values. The sparse 1x1 convolutions are defined by a sparse C×C' weight matrix, which has multiple empty weights equal to zero and multiple non-empty weights. The input channels form a dense C'×HW activation matrix, which has eigenvalues ​​defined for each element of the activation matrix. The system includes a non-transitory computer storage medium and a processing unit, the non-transitory computer storage medium storing instructions that, when executed by the processing unit, cause the processing unit to: Obtain an indication of the empty weights of the sparse C×C' weight matrix stored in the first memory cell; and The sparse C×C' weight matrix is ​​processed by combining the dense C'×HW activation matrix with the following steps: Convolution values ​​of the multiple elements in the row vectors containing the multiple elements of the dense C'×HW activation matrix are generated from the elements of the row vectors. (a) Extracting the corresponding feature values ​​of the input channel from the second memory cell storing the dense C'×HW activation matrix, wherein the extracted corresponding feature values ​​are non-empty weights of the sparse C×C' weight matrix for these feature values, according to the instruction; and (b) Form a corresponding sum of the extracted corresponding feature values ​​weighted by the corresponding non-empty weights.

15. The system according to claim 14, wherein, The system is configured to implement the method according to any one of claims 1 to 11.

16. A non-transitory computer-readable storage medium storing computer program instructions for implementing the method according to any one of claims 1 to 13.

Images