Calculation device, calculation method, and program

Abstract

The invention relates to a calculation device, a calculation method, and a program. This computing device (10) computes a matrix product of a sparse matrix and a weight matrix in a neural network, the sparse matrix having a structure in which a predetermined number of non-zero elements are included in each block of a predetermined size. The sparse matrix is defined by a local index representing the position of the non-zero element in each block and the value of the non-zero element, and the arithmetic device (10) performs: a process for acquiring, from the weight matrix, an element corresponding to the position of the non-zero element in the sparse matrix by referring to the local index; and multiplication and accumulation operation of non-zero elements of the sparse matrix and corresponding elements of the weight matrix is carried out.

Description

TECHNICAL FIELD

[0001] The present application relates to an operation device, an operation method, and a program of a neural network. BACKGROUND

[0002] Since the power and the operation performance of an in-vehicle AI processor are limited, a light-weight deep neural network (DNN) model is sought. A DNN is constituted by the multiplication of a learned weight and a feature map inside the network. In general, a wide DNN having a large number of channels of the feature map has a good performance, but it is difficult to run on a hardware having a small resource such as an in-vehicle machine because the operation amount increases.

[0003] It is known that the number of product-sum operations can be reduced while maintaining the performance by learning a wide DNN so that the feature map becomes sparse and the operation related to zero is omitted (Non-Patent Literature 1).

[0004] Non-Patent Literature 1: Mark Kurtz et al., “Inducing and exploiting activation sparsity for fast inference on deep neural networks.”, 119:5533-5543, 2020.

[0005] The operation method described in Non-Patent Literature 1 is difficult to perform efficient operation on a vector-type operation device such as a widely spread GPU because it is to omit the operation of an irregular zero pattern. An object of the present disclosure is to realize a DNN model having a sparse feature representation and having good compatibility with a vector-type operation device, and further to realize an operation with good power efficiency. SUMMARY

[0006] The operation device of the present disclosure is an operation device that performs matrix multiplication of a sparse matrix and a weight matrix in a neural network, the sparse matrix having a configuration in which a prescribed number of non-zero elements are present in each block of a prescribed size, the sparse matrix being defined by a local index indicating the position of the non-zero element within each block and the value of the non-zero element, the operation device performing the following processing: referring to the local index, acquiring an element corresponding to the position of the non-zero element of the sparse matrix from the weight matrix, and performing product-sum operation of the non-zero element of the sparse matrix and the corresponding element of the weight matrix. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a diagram showing an outline of the operation processing performed by the operation device of the embodiment.

[0008] Figure 2is a diagram showing an example of a sparse matrix processed by the operation device of the embodiment.

[0009] Figure 3 is a diagram showing an example of a local index.

[0010] Figure 4 is a diagram showing a method of mapping an input matrix to a space of a sparse matrix.

[0011] Figure 5 is a diagram showing a process of a matrix product of a sparse matrix and a weight matrix.

[0012] Figure 6 is a diagram showing a structure of a vector operation device of the embodiment.

[0013] Figure 7 is a flowchart showing an action based on an operation process of the operation device.

[0014] Figure 8 is a diagram showing an example of a matrix product with a weight matrix by a conventional technology.

[0015] Figure 9 is a diagram showing a result of comparing precision using CIFAR-10.

[0016] Figure 10 is a diagram showing a result of comparing precision using CIFAR-100.

[0017] Figure 11 is a diagram showing a result of comparing precision using ImageNet. DETAILED DESCRIPTION

[0018] The operation device of the embodiment is an operation device that performs an operation of a matrix product of a sparse matrix and a weight matrix in a neural network, the sparse matrix having a configuration of having N non-zero elements in each block of a size of M elements (M>N), the sparse matrix being defined by a local index showing positions of the non-zero elements within each block and values of the non-zero elements, the operation device performing a process of acquiring elements corresponding to positions of non-zero elements of the sparse matrix from the weight matrix with reference to the local index, and performing a multiplication-accumulation operation of the non-zero elements of the sparse matrix and the corresponding elements of the weight matrix.

[0019] By using a sparse matrix having a configuration in which N out of M elements are nonzero in this way, it is possible to operate a neural network that actually has a wider feature representation with the same number of product-sum operations as a narrow neural network that has only 1 / M of the channels. That is, it is possible to suppress the operation load and improve the expressiveness of the neural network. In addition, since the sparse matrix always has only N elements with values within a block, it is possible to perform operations in parallel per block, and it is possible to perform parallel operations on a vector-type operation unit like a GPU.

[0020] Furthermore, the operation device of the embodiment can also be a vector-type operation device. That is, the operation device of the embodiment can also be a vector-type operation device that has a plurality of kernels and performs matrix multiplication of a sparse matrix and a weight matrix in a neural network, wherein the sparse matrix has a configuration in which N nonzero elements are present in each block of a size in which the number of elements is M, the sparse matrix is defined by local indices that indicate the positions of the nonzero elements within each block and the values of the nonzero elements, the plurality of kernels refer to the local indices to acquire elements corresponding to the positions of the nonzero elements of the sparse matrix from the weight matrix for different blocks, and perform, in parallel, processing of product-sum operations of the nonzero elements of the sparse matrix and the corresponding elements of the weight matrix, while any one of the kernels performs processing of aggregating the results of the product-sum operations obtained by the plurality of kernels.

[0021] The operation device of the embodiment can also accept input of an input matrix of which the rank is lower than the sparse matrix, and perform mapping processing that maps the input matrix to the space of the sparse matrix and generates local indices that indicate the positions of the elements of the input matrix that are mapped.

[0022] In the embodiment, a sparse matrix having a configuration is realized by mapping an input matrix of which the rank is lower than the sparse matrix to the space of the sparse matrix. For example, if the weight matrix is to be made sparse, it is possible to make elements of which the values obtained by learning are small to be zero (prune the small values), but in the case of the input matrix, since the values are not known in advance, it is not possible to adopt the same method. Mapping the input matrix of which the rank is low to a high-dimensional space and generating a sparse matrix is a design for sparsifying the input matrix.

[0023] The operation device of the embodiment can also determine, in the mapping processing, the mapping destination of an element of the input matrix based on the sign of the element and the signs of elements adjacent to the element.

[0024] The operation device of the embodiment can also use, in the mapping processing, the minimum value among a plurality of elements adjacent to an element that should be mapped, as the value of the element that is mapped to the space. By this, it is possible to suppress the variation in the values of the nonzero elements of the sparse matrix with respect to the variation in the values of the elements.

[0025] In the operation device of the embodiment, the non-zero elements of the sparse matrix described above can also be quantized data. The operation device of the embodiment can suppress the operation amount by using the sparse matrix, but the number of parameters increases compared to a dense matrix having only non-zero elements. By quantizing the data, the amount of memory usage can be reduced, and the disadvantage of an increase in the number of parameters can be mitigated. Furthermore, when using quantized data, the accuracy can correspondingly decrease, but the operation device of the embodiment has a larger feature dimension compared to a dense matrix, and thus the accuracy can be recovered. By using quantized data in this way, a neural network that suppresses the disadvantage of an increase in parameters and has high performance can be implemented.

[0026] The operation device of the embodiment can also adjust the learning rate for each element based on the number of updates of each element of the weight matrix and perform learning in the process of performing learning of the weight matrix described above.

[0027] In the present embodiment, since a sparse matrix is used, the elements of the weight matrix corresponding to the zero elements of the weight matrix are not updated at the time of learning. That is, learning of each element of the weight matrix is not uniformly performed, and the number of updates differs depending on the element. In learning in which learning of all elements is uniformly performed, learning cannot be performed well. In the present embodiment, by adjusting the learning rate for each element based on the number of updates of each element of the weight matrix and performing learning, learning can be appropriately performed.

[0028] The operation method of the embodiment is an operation method of performing, by an operation device, an operation of matrix multiplication of a sparse matrix and a weight matrix in a neural network, in which the sparse matrix has a configuration in which a prescribed number of non-zero elements are present in each block of a prescribed size, the sparse matrix is defined by a local index indicating the positions of the non-zero elements within each block and the values of the non-zero elements, and the operation method includes a step in which the operation device acquires, with reference to the local index, elements of the weight matrix corresponding to the positions of the non-zero elements of the sparse matrix, and a step in which the operation device performs a multiplication-accumulation operation of the non-zero elements of the sparse matrix and the corresponding elements of the weight matrix.

[0029] The operation method of the embodiment can also include a step in which the operation device accepts input of an input matrix having a lower rank than the sparse matrix described above, maps the input matrix to the space of the sparse matrix, and generates a local index indicating the positions of the elements of the input matrix that have been mapped.

[0030] The implementation program is for performing matrix multiplication of a sparse matrix and a weight matrix in a neural network, wherein the sparse matrix is ​​constructed with a predetermined number of non-zero elements in each block of a predetermined size, and the sparse matrix is ​​defined by local indices representing the positions of the non-zero elements in each block and the values ​​of the non-zero elements. The computer performs the following steps: obtaining the element corresponding to the position of the non-zero element of the sparse matrix from the weight matrix with reference to the local index; and performing a multiplication and summation operation of the non-zero elements of the sparse matrix and the corresponding elements of the weight matrix.

[0031] The program of the implementation method can also enable the computer to perform the following steps: accepting an input matrix with a rank lower than the sparse matrix, and performing a mapping process that maps the input matrix to the space of the sparse matrix and generates local indices representing the positions of the elements of the mapped input matrix.

[0032] Hereinafter, the computing device of this embodiment will be described with reference to the accompanying drawings. Furthermore, the following description shows only one example of a preferred embodiment and is not intended to limit the invention as described in the claims.

[0033] Figure 1 This is a diagram illustrating the general computational processing performed by the computing device in the embodiment. The computing device performs a matrix multiplication operation on the input matrix and the weight matrix. The rank of the input matrix is ​​lower than that of the sparse matrix. Figure 1 In the example shown, the input matrix is ​​4 rows and 4 columns, the sparse matrix is ​​16 rows and 4 columns, and the weight matrix is ​​12 rows and 16 columns. Furthermore, the number of rows and columns of the input matrix, sparse matrix, and weight matrix processed by the computing device is not limited to these values. Figure 1 The example shown. The matrix processed by the computing device can also be a matrix consisting of 1 row or 1 column.

[0034] An example of an input matrix is ​​a feature map, which is the activation output of a layer in a convolutional neural network (CNN). Here, activation refers to the output after passing through an activation function in the neural network. Furthermore, the computing apparatus disclosed herein is not limited to CNNs, but can also be applied to activation matrices of other DNNs such as multilayer perceptrons (MLPs) and attention mechanisms.

[0035] The processing unit maps the input matrix to a space of sparse matrices. Figure 1 In the example shown, a 4x4 input matrix is ​​mapped to a 16x4 space. This mapping maps the values ​​of each element of the input matrix to elements of the sparse matrix, forming the non-zero elements of the sparse matrix. Elements of the input matrix that are not mapped form the zero elements of the sparse matrix.

[0036] Figure 2 This is a diagram illustrating an example of a sparse matrix processed by the computing device of this embodiment. For example... Figure 2 As shown, the sparse matrix has blocks B1, B2, B3, B4, etc., consisting of four elements. Figure 2 Although not all blocks are labeled with the attached diagram, the sparse matrix has 16 blocks.

[0037] exist Figure 2 In the sparse matrix shown, one element in a block of four consecutive elements is a non-zero element. Figure 2 Elements with shaded areas are non-zero elements, and elements without shaded areas are zero elements. The numbers 1 through 4 recorded in the four elements of block B1 are not the values ​​of the elements, but rather indicate which element within the block it is. In block B1, the first element is a non-zero element. That is, the element at (1, 1) of the input matrix is ​​mapped to the first element of block B1. The second through fourth elements of block B1 are zero elements.

[0038] Thus, the sparse matrix does not have randomly distributed non-zero elements; rather, it follows the rule that each block contains one non-zero element, and therefore possesses a certain construction. Furthermore, in Figure 2 The example given is that the size of the block consists of four elements, one of which is a non-zero element. However, more generally, the sparse matrix of this disclosure has the construction of N (M>N) non-zero elements in a block consisting of M consecutive elements.

[0039] Back Figure 1 The processing unit maps the input matrix to the space of sparse matrices and generates a sparse matrix, which is then managed using local indices. A local index is data that indicates which of the four elements in each block of the sparse matrix is ​​a non-zero element.

[0040] Figure 3 This is a diagram illustrating an example of a local index. Figure 3 The example shown corresponds to Figure 2 The sparse matrix shown is such that index In1 is "1", which indicates that the first element in block B1 is a non-zero element. Furthermore, the values ​​of the non-zero elements can be determined by referring to the values ​​of the original input matrix. For example, the value of the non-zero element indicated by index In1 is the value of the element at (1, 1) of the input matrix.

[0041] In this way, the sparse matrix is ​​managed by a combination of the original input matrix (here, a non-zero matrix) and local indices, without needing to have the values ​​of each element in a 16-row, 4-column array as data. Figure 1 The 16-row, 4-column diagram shown is a conceptual diagram for illustration. By using local indexes, memory capacity can be reduced compared to storing the sparse matrix itself.

[0042] Here, we will explain the term "non-zero element". A "non-zero element" is an element that is designated as non-zero by a local index. Since the elements of the input matrix are mapped to non-zero elements, a non-zero element is assigned a value of zero when the elements of the input matrix are exactly zero, but this case is also called a non-zero element.

[0043] Figure 4 This is a diagram illustrating the method of mapping an input matrix to a sparse matrix space. The computational device in this embodiment is based on element X. j and the element X that follows it j＋1 The sign of the element X in the mapping input matrix is ​​used to determine the element X. j The location. Figure 4 The matrix on the left represents a portion of the input matrix, with elements X0, ..., X3 consecutive from top to bottom. Figure 4 The central graphic representation element X j and element X j＋1 The relationship between the sign and the position to be mapped. The horizontal axis represents X. j The symbols are arranged with positive on the right and negative on the left. The vertical axis represents X. j＋1 The symbol is , with the top representing positive and the bottom representing negative.

[0044] In X j positive and X j＋1 When it is positive, the element X corresponds to the "1" in the upper right region. j It is mapped to the first one within the block. In X j positive and X j＋1 In the case of a negative value, the element X corresponds to the "2" in the lower right region. j It is mapped to the second one within the block. In X j negative and X j＋1 In the case of a negative value, the element X corresponds to the "3" in the lower left region. j It is mapped to the third one within the block. In X j negative and X j＋1 When positive, element X corresponds to "4" in the upper left region. j The fourth element is mapped into the block. Thus, the input matrix is ​​used as a function to perform a mapping from the input matrix to the sparse matrix space based on the contents of the input matrix (in this case, the input activations). Figure 4 The matrix on the right is a sparse matrix, and the elements of the input matrix are mapped one by one to the blocks of the sparse matrix.

[0045] Figure 5 This is a graph representing the processing of the matrix product of a sparse matrix and a weight matrix. Figure 5 In China, with Figure 1The example shown is the cumulative summation of the multiplication of the first row R1 of the weight matrix and the first column C1 of the sparse matrix. To read the value of the first column C1 of the sparse matrix, the data of the first column In of the local index is used. The processing unit reads the elements of the weight matrix located at the positions corresponding to the non-zero elements of the sparse matrix. Specifically, it obtains the four elements indicated by the local index In from the 16 elements of the first row R1 of the weight matrix. The processing unit multiplies the four obtained elements of the weight matrix with the four elements of the first row of the sparse matrix respectively, and then adds the products. Thus, the cumulative summation of the multiplication of the first row R1 of the weight matrix and the first column C1 of the sparse matrix is ​​obtained.

[0046] Figure 6 This diagram illustrates the structure of the computing device 10 according to the embodiment. The computing device 10 is a vector-type computing device, including a CPU 11, a GPU 12, and a storage unit 13. The computing device 10 is connected to a data input unit 20 and an output unit 21. The input unit 20 has the function of inputting data to the computing device 10, and the output unit 21 has the function of outputting the results of calculations based on the computing device 10.

[0047] GPU 12 has multiple cores, enabling it to execute processing tasks in parallel. Storage unit 13 stores programs for executing the processing of computing device 10 and neural network models. CPU 11 and GPU 12 implement the functions of computing device 10 by reading and executing the programs stored in storage unit 13. Figure 7 The actions shown.

[0048] Figure 7 It is a flowchart showing the operation of computation processing based on the arithmetic device 10. Figure 7 The diagram illustrates the operation of calculating the matrix product of the input matrix and the weight matrix for one layer of a neural network. The processing unit 10 acquires the input matrix (S10). The input matrix is, for example, the activation matrix output from the previous layer.

[0049] The processing unit 10 maps the input matrix to the space of sparse matrices (S11). For example... Figure 4 As explained, the arithmetic unit 10 determines the mapping position within each block based on the sign of each element of the input matrix, thereby generating a structured sparse matrix (conceptually, for example, referring to...) where each block contains one non-zero element. Figure 2 ).

[0050] The processing unit 10 performs matrix multiplication on the sparse matrix and the weight matrix (S12). Specifically, multiple cores of the GPU 12 are used to execute the multiplication and accumulation operations of each block in parallel. For example, the multiplication and accumulation operations of blocks B1 to B4, blocks B5 to B8, blocks B9 to B12, and blocks B13 to B16 are executed by separate cores.

[0051] Each kernel of the processing unit 10 performs a reference local indexing operation to obtain the index of the element corresponding to the non-zero element from the weight matrix (S13), and performs a multiplication and accumulation operation of the obtained elements of the weight matrix and the elements of the sparse matrix (S14). Thus, the processing unit 10 calculates the value of each element of the output matrix. For example, the processing unit 10 calculates the output matrix for the next layer output (S15).

[0052] The above describes the computing device 10 and computing method of this embodiment. The computing device 10 of this embodiment maps the input matrix to a structured sparse matrix and calculates the matrix product of the sparse matrix and the weight matrix, thereby improving the expressive power of the neural network.

[0053] Figure 8 The diagram illustrates an example of performing a matrix product with a weight matrix on a 4x4 input matrix without mapping it to a sparse matrix. Figure 8 As shown, to calculate the matrix product with the 4x4 input matrix, the weight matrix is ​​limited to 4 columns. (Comparison) Figure 1 and Figure 8 It can be clearly seen that in this embodiment, by mapping the input matrix to a sparse matrix, a larger weight matrix can be used, thereby improving the expressive power of the neural network.

[0054] Furthermore, the sparse matrix used in this embodiment is a structured sparse matrix with a fixed number of non-zero elements in each block. Since the positions of non-zero elements are managed as local indices, the multiplication and addition operations of zero elements can be omitted through indexing operations, and the computation can be performed with the same load as the matrix multiplication operation of the input matrix before sparsification. In addition, since the computation can be performed for each block, parallel processing in GPU12 can be achieved, thereby reducing the processing time.

[0055] Next, the learning of the weight matrix used in the computing device 10 of this embodiment will be explained. The learning of the weight matrix follows the same approach as that used in neural network models such as Adam (Diederik P Kingma and Jimmy Ba. Adam “A method forstochastic optimization.” Dec. 2014.) and AdamW (Ilya Loshchilov and Frank Hutter. “Decoupled weight decay regularization.” In International Conference on Learning Representations, 2019.). However, in the learning of the weight matrix in this embodiment, a design is adopted that adjusts the learning rate for each element. Specifically, the learning rate is adjusted based on the number of times each weight element is updated.

[0056] Adam and AdamW use an adaptive learning rate based on statistical values ​​of gradient history, but it is known that this adaptive learning rate can become unstable in the initial stages of learning when the number of experienced gradients is small. In the neural network of this embodiment, since each weight element receives gradients less frequently than in a denser network (on average, 1 / M times), this problem may become significant. In this embodiment, a method that modifies the formalization of RAdam is proposed. In this specification, this proposed method is referred to as "ERAdam (Experience-RAdam)".

[0057] First, let's explain RAdam. RAdam reduces variance in the initial learning phase by multiplying the adaptive learning rate of Adam or AdamW by rt ​​< 1.

[0058] [Formula 1]

[0059]

[0060] Here, t is the number of optimization steps, and β is the hyperparameter used when calculating the running average of the gradient. As t→∞ (learning progress), rt→1, and the original Adam or AdamW update is resumed.

[0061] The ERAdam proposed in this embodiment scales the number of steps t in calculating rt for each weight element to be proportional to the number of gradients received. Specifically, instead of t, a scaled time vector t∈R is used as follows: C￣i .

[0062] [Equation 2]

[0063]

[0064] In this implementation, the optimizer is combined with a scheduler that has k-decay (Tao Zhang and Wei Li. kdecay “Just adding k-decay items on learning-rate schedule to improve neural networks” arXiv preprint arXiv:2004.05909, 2020.) and CosineDecay (Ilya Loshchilov and Frank Hutter. “SGDR: Stochastic gradient descent with warm restarts.” In International Conference on Learning Representations, 2017.).

[0065] The above description illustrates the computing device and computing method of this disclosure using examples of embodiments, but this disclosure is not limited to the above embodiments.

[0066] The above implementation illustrates an example of performing matrix multiplication of the sparse matrix and the weight matrix after mapping the input matrix to a sparse matrix. However, a structured sparse matrix can also be used as the input matrix. By using local indexing to perform matrix multiplication of the structured sparse matrix and the weight matrix, the computational complexity can be reduced and the expressive power of the neural network can be improved.

[0067] The computing device disclosed herein can also use quantized data as non-zero elements of a sparse matrix. By using quantized data, the computational load can be reduced.

[0068] The computational apparatus of this disclosure can also use the minimum value among multiple elements adjacent to the element to be mapped when mapping an input matrix to the space of a sparse matrix. In the computational apparatus of this disclosure, since the input matrix is ​​used as a function to determine the mapping destination, the continuity between the input matrix and the sparse matrix may be impaired. By using a smaller value as the element to be mapped, changes in the values ​​of non-zero elements can be suppressed in response to changes in the values ​​of individual elements.

[0069] Example

[0070] In the embodiments, a comparison was made of the expressive power of using sparse matrices versus not using them, and a comparison was made of the method of the present disclosure that makes the input matrix sparse with the method that makes the weight matrix sparse.

[0071] The classification accuracy was evaluated using CIFAR-10 (Alex Krizhevsky et al., “Cifar-10” Canadian institute for advanced research), CIFAR-100 (Alex Krizhevsky et al., “Cifar-100” Canadian institute for advanced research), and ImageNet (Jia Deng et al., “ImageNet: A large-scale hierarchical image database.” In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, June 2009.). Since commonly used networks already achieve good saturation accuracy, a narrower version was used as the base network to better evaluate the effect of using sparse activations and weights. Specifically, a width α times narrower (W∈R) was used except for the first and final layers. Co / α×C￣i / α The network consumes approximately α less. 2 The multiple count is multiple. ReLU, SWS (Structured Weight Sparsification), and the network of this disclosure (hereinafter referred to as "SAS (Structured Activation Sparsification)") have the same multiple count for the same α. Here, SWS is a technique for sparsifying the weight matrix instead of activation, as described in Aojun Zhou et al., "Learning N:M fine-grained structured sparse neural networks from scratch" Feb. 2021.

[0072] To ensure a fair comparison, the same learning strategy was employed for all variables, namely, the number of learning rounds, batch size, optimizer, scheduler, etc. The ERAdam optimizer described in this embodiment was used for all variables. Since ∇X in equation (1) above is almost identical for each weight element, it is equivalent to RAdam for the base network. The reason for using the method of Aojun Zhou et al. in the learning of the SWS network is that the codebase supports arbitrary 1:M sparsity and enables learning from scratch that can be fairly compared with SAS. In this embodiment, the network was trained from scratch in all experiments using the data used in the ConvNeXt paper (Zhuang Liu et al., “A convnet for the 2020s.” In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition pages 11976-11986, 2022.). The main experimental configurations are summarized in the table below.

[0073] [Table 1]

[0074]

[0075] In CIFAR-10 and CIFAR-100, ResNet18, one of the most commonly used architectures, was used (Kaiming He et al., “Deep residual learning for image recognition.”In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.). Figure 9 This is a graph representing the results using CIFAR-10. Figure 10 This graph represents the results using CIFAR-100. The method from the ConvNeXt paper was used in ImageNet. ConvNeXt is computationally efficient due to its network architecture design. For example, it employs a large number of depth-oriented convolutions to reduce overlap. Figure 11 This is a graph representing the results of using ImageNet.

[0076] exist Figures 9-11 In the diagram, the vertical axis represents correctness, and the horizontal axis represents the experimental conditions. The ReLU on the horizontal axis represents the result of the network without sparsification; M=2, M=4, M=8, and M=16 represent the results after M-fold sparsification. The solid line represents the SAS result, and the dashed line represents the SWS result.

[0077] This comparative experiment shows that increasing the sparsity M of the activations improves accuracy. A maximum gain was observed across the entire range of the base network width (with different α values), from the base ReLU network to the sparse network with M=2. Improvement saturates at M=8 or M=16. Similarly, a larger gain (larger α) was observed with a smaller base network capacity (larger α).

[0078] Although the SWS network also shows improvement for weight sparsity M, the accuracy of SAS is significantly higher than that of SWS with the same sparsity M.

[0079] SAS improves accuracy without increasing multi-counting and actual inference time across wide network constructions and diverse datasets. The increased flexibility resulting from the increase in weights is considered the main factor contributing to the accuracy improvement of SAS compared to SWS (for the same computational cost).

Claims

1. An arithmetic device that performs an operation of a matrix product of a sparse matrix and a weight matrix in a neural network, wherein the sparse matrix has a configuration in which a prescribed number of non-zero elements are present in each block of a prescribed size, the sparse matrix is defined by local indices that indicate positions of the non-zero elements within the respective blocks and values of the non-zero elements, and the arithmetic device performs the following processing: acquires elements corresponding to the positions of the non-zero elements of the sparse matrix from the weight matrix with reference to the local indices, and performs a product-sum operation of the non-zero elements of the sparse matrix and the corresponding elements of the weight matrix.

2. The arithmetic device according to claim 1, wherein an input matrix of a lower rank than the sparse matrix is input, and a mapping process that maps the input matrix to a space of the sparse matrix and generates local indices that indicate positions of elements to which the input matrix is mapped is performed.

3. The arithmetic device according to claim 2, wherein in the mapping process, a mapping destination of an element of the input matrix is determined on the basis of a sign of the element and signs of elements adjacent to the element.

4. The arithmetic device according to claim 2, wherein in the mapping process, as a value of an element mapped to the space, a minimum value among a plurality of elements adjacent to the element that should be mapped is used.

5. The arithmetic device according to claim 1 or 2, wherein the non-zero elements of the sparse matrix are quantized data.

6. The arithmetic device according to claim 1 or 2, wherein in a process of learning the weight matrix, a learning rate is adjusted for each element on the basis of a number of updates of each element of the weight matrix, and learning is performed.

7. An arithmetic method that performs an operation of a matrix product of a sparse matrix and a weight matrix in a neural network by an arithmetic device, wherein the sparse matrix has a configuration in which a prescribed number of non-zero elements are present in each block of a prescribed size, the sparse matrix is defined by local indices that indicate positions of the non-zero elements within the respective blocks and values of the non-zero elements, and the arithmetic method comprises: a step in which the arithmetic device acquires elements corresponding to the positions of the non-zero elements of the sparse matrix from the weight matrix with reference to the local indices, and a step in which the arithmetic device performs a product-sum operation of the non-zero elements of the sparse matrix and the corresponding elements of the weight matrix.

8. The arithmetic method according to claim 7, wherein the arithmetic device receives input of an input matrix of a lower rank than the sparse matrix, maps the input matrix to a space of the sparse matrix, and generates local indices that indicate positions of elements to which the input matrix is mapped.

9. A program for performing a product-sum operation of a sparse matrix and a weight matrix in a neural network, wherein the sparse matrix has a configuration in which a prescribed number of non-zero elements are present in each block of a prescribed size, the sparse matrix is defined by local indices that indicate positions of the non-zero elements within the respective blocks and values of the non-zero elements, and the program causes a computer to perform the following steps: a step in which the computer acquires elements corresponding to the positions of the non-zero elements of the sparse matrix from the weight matrix with reference to the local indices, and a step in which the computer performs a product-sum operation of the non-zero elements of the sparse matrix and the corresponding elements of the weight matrix. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ a step of acquiring elements corresponding to positions of non-zero elements of the sparse matrix from the weight matrix with reference to the local index; and a step of performing a product-summation operation of non-zero elements of the sparse matrix and corresponding elements of the weight matrix.

10. The program according to claim 9, wherein, a step of causing the computer to perform input of an input matrix of which rank is lower than that of the sparse matrix, map the input matrix to a space of the sparse matrix, and generate a local index indicating positions of elements to which the input matrix is mapped.

Images