Calculation device, calculation method, and calculation program

Abstract

To provide a computing device capable of processing calculations appropriately. [Solution] The arithmetic unit 10 calculates the matrix product of a first matrix and a second matrix. The arithmetic unit 10 includes a first arithmetic unit 30 that calculates the first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution which is the distribution of numerical values ​​stored in the first matrix; a second arithmetic unit 40 that can operate in parallel with the first arithmetic unit 30 in time, and calculates the second separation matrix product, which is the matrix product of the first matrix and the second matrix, where a second separation matrix is ​​separated from the first matrix where numerical values ​​different from those stored in the first separation matrix are separated; and a matrix addition unit 50 that adds the first separation matrix product and the second separation matrix product to obtain the matrix product of the first matrix and the second matrix.

Description

[Technical Field]

[0001] The disclosure in this specification relates to a technique for performing matrix multiplication operations. [Background technology]

[0002] Non-patent document 1 describes a method for performing the matrix product of a neural network's feature matrix and weight matrix. Specifically, the weight matrix is ​​obtained by first adding the dense matrix and the low-rank matrix before obtaining the matrix product, and then saving the result. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tim Dettmers, Artidoro Pagnoni, Ari HoltzmanLuke Zettlemoyer “QLORA: Efficient Finetuning of Quantized LLMs” arXiv preprint arXiv:2305.14314v1 2023 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the resulting matrix has a large number of bits, and the matrix multiplication operation tends to be computationally intensive. Therefore, there are concerns that the hardware required for processing will be larger and the processing speed will be slower. Consequently, it is necessary to suppress the decrease in calculation accuracy and processing speed, and to execute the calculations appropriately.

[0005] One of the purposes of this specification disclosure is to provide a calculation device, a calculation method, and a calculation program capable of appropriately processing calculations. [Means for solving the problem]

[0006] One aspect disclosed herein is a arithmetic device for performing the matrix product of a first matrix and a second matrix, A first arithmetic unit (30) performs a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution which is the distribution of numerical values ​​stored in the first matrix. The first arithmetic unit is an arithmetic unit that can perform operations in parallel over time, and includes a second arithmetic unit (40) that performs the matrix product of the second separation matrix, which is the matrix product of the second separation matrix and the second separation matrix, where the second separation matrix is ​​obtained by separating the numerical values ​​from the first separation matrix from the first separation matrix. The system includes a matrix addition unit (50) that adds the product of a first separation matrix and the product of a second separation matrix to obtain the matrix product of the first matrix and the second matrix.

[0007] Another aspect of the disclosed embodiment is a method for performing the matrix product of a first matrix and a second matrix, A first operation is performed by performing a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution that is the distribution of the numerical values ​​stored in the first matrix. The first operation is performed in parallel in time, and the second operation is the matrix product of the second separation matrix, which is the matrix product of the second separation matrix (which stores all values ​​except those stored in the first separation matrix) and the second input matrix. This includes matrix addition, which involves adding the product of a first separable matrix to the product of a second separable matrix to obtain the matrix product of the first matrix and the second matrix.

[0008] Another aspect of the disclosed embodiment is a computation program that causes a processing unit (90a) capable of processing multiple processes in parallel in time to perform a matrix product of a first matrix and a second matrix, In the processing unit, A first operation is performed by performing a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution that is the distribution of the numerical values ​​stored in the first matrix. The first operation is performed in parallel with the second operation, which is the matrix product of the second matrix, where the second matrix contains all values ​​except those stored in the first matrix, and the second matrix. matrix addition for adding the first separated matrix product and the second separated matrix product to obtain a matrix product of the first matrix and the second matrix; and causing said processing to be executed.

[0009] According to these aspects, in a state where the first matrix is separated into the first separated matrix and the second separated matrix, matrix products with the second matrix are calculated in parallel with each other in time. This can suppress an increase in the calculation scale of a single matrix product, and increase the calculation processing speed through temporal parallel processing. Furthermore, since calculation can be executed with optimization respectively applied to the calculation of the first separated matrix product and the calculation of the second separated matrix product in accordance with the separation form based on the numerical value distribution between the first separated matrix and the second separated matrix, it is possible to contribute to suppressing a decrease in calculation accuracy. As described above, calculation can be processed appropriately.

[0010] It should be noted that reference signs in parentheses included in the claims and the like exemplify the correspondence with parts of the embodiments described later, and are not intended to limit the technical scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] [Figure 1] Figure schematically showing a hardware configuration of an arithmetic device. [Figure 2] Figure schematically showing an example of a configuration of a dedicated circuit. [Figure 3] Timing chart showing temporal relationships of processing. [Figure 4] Figure for explaining symmetric quantization. [Figure 5] Figure for explaining asymmetric quantization. [Figure 6] Figure schematically showing an example of a configuration of a dedicated circuit. [Figure 7] Figure explaining an example of use of a mask in sparse matrix multiplication. [Figure 8] Figure explaining an example of use of a mask in sparse matrix multiplication. [Figure 9] Timing chart explaining time-division processing. [Figure 10] Figure schematically showing a hardware configuration of an arithmetic device. [Figure 11] A flowchart illustrating the processing method used by the arithmetic unit. [Modes for carrying out the invention]

[0012] Several embodiments will be described below with reference to the drawings. In each embodiment, the same reference numerals are used for corresponding components, and redundant explanations may be omitted. If only a part of the configuration is described in each embodiment, the configuration of other embodiments described earlier can be applied to the other parts of that configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations from multiple embodiments can be partially combined even if not explicitly stated, as long as there are no particular problems with the combination.

[0013] (First Embodiment) The computing device 10 of the first embodiment shown in Figure 1 is used for neural network computation. The computing device 10 may be mounted on a moving object such as a vehicle and used for neural network computation in the moving object. The neural network may be used for obstacle detection by analyzing sensor data from a surrounding monitoring sensor that monitors the area around the vehicle. Here, the sensor data may be images taken by a camera acting as a surrounding monitoring sensor, and the neural network may recognize targets captured in the images. Alternatively, the neural network may be used for route planning in the autonomous driving of a vehicle.

[0014] The computing unit 10 may perform only the computational processing of the neural network, or it may perform the computational processing of the neural network and specific processing to realize the functions of the mobile body, including computations by other computer programs, etc. The computing unit 10 may be a so-called ECU (Electronic Control Unit), or it may be a dedicated device specialized for the computational processing of the neural network.

[0015] In the following description, the arithmetic unit 10 is an ECU that performs specific processing to realize the functions of the mobile body, including calculations performed by other computer programs, etc.

[0016] The arithmetic unit 10 is primarily composed of a computer. The computer comprising the arithmetic unit 10 has at least one CPU (Central Processing Unit) 10a and at least one memory 10b. The memory 10b is a non-transitional, tangible storage medium that non-temporarily stores computer programs and data that can be read by the CPU 10a. Furthermore, the computer may have a rewritable, volatile storage medium such as RAM (Random Access Memory) 10c. The RAM 10c is used for temporary storage of data being processed. The CPU 10a is a processor that can execute various processes according to the computer programs stored in the memory 10b. Furthermore, the computer includes an interface 10d for communicating data with the outside world.

[0017] Furthermore, the computing unit 10 has at least one dedicated circuit 20 for executing at least a portion of the neural network's computational processing. The dedicated circuit 20 may be, for example, a GPU-based circuit, a custom integrated circuit of an ASIC, or another type of circuit. The dedicated circuit 20 may also be composed of a combination of multiple types of circuits.

[0018] Here, we will describe in detail the neural network that is the target of the computational processing of the dedicated circuit 20. The structure of the neural network described below may be implemented by the hardware of the dedicated circuit 20. The structure of the neural network may also be stored as a computer program in memory 10b and implemented by software. Furthermore, the structure of the neural network may be implemented by a combination of hardware and software.

[0019] A neural network is an algorithm modeled after the human brain and is also known as artificial intelligence. A neural network consists of multiple layers. These layers are composed of multiple nodes called artificial neurons, and each neuron is connected to neurons in the preceding and succeeding layers. The strength of these connections is called a weight.

[0020] Multiple layers may include, for example, pooling layers and fully connected layers, in addition to convolutional layers. Each convolutional layer performs an operation by convolving a local filter (a kernel or feature extractor) of a predetermined size defined for each convolutional layer onto the input data (features). For example, the input value entered at the position corresponding to the neuron in the kernel is multiplied by a weight pre-set in the kernel for each neuron, and a linear sum of these multiplicative values ​​is calculated. The convolution operation is performed by adding a bias pre-set in the kernel to this linear sum. The result of such a convolution operation may be transformed using, for example, a ReLU (Rectified Linear Unit) activation function. Furthermore, the convolutional layer can be multilayered to perform the convolution operation and transformation by the activation function multiple times.

[0021] The pooling layer calculates local statistics of the feature map output from the convolutional layer, which is a higher layer connected to the input. Specifically, it sets a window of a predetermined size (e.g., 2x2, 3x3) corresponding to the position of the higher layer, and calculates local statistics from the input values ​​within the window. For example, the maximum value may be used as the local statistic. That is, by adopting the maximum value of the input values ​​within the window as the result of pooling, the input values ​​within the window can be integrated. Unlike the kernel set for the convolutional operation, the windows set for this pooling operation often do not overlap in position (although some overlap is permitted). As a result, the size of the feature map output from the pooling layer is reduced according to the size of the window. This is called downsampling.

[0022] A fully connected layer is a layer in which each neuron is connected to all neurons in the higher layer connected to the input side, and is mainly used for feature extraction, vector representation transformation, etc. While fully connected layers are relatively easy to implement, they also have the challenge of requiring a large amount of computation due to the large number of parameters.

[0023] The dedicated circuit 20 is capable of performing the operations used in such neural networks. Specifically, the dedicated circuit 20 performs matrix multiplication operations as shown in Equation 1 below.

[0024]

number

[0025] If the product of W and X is processed directly, the amount of matrix multiplication processing required will be large because W is large. Therefore, instead of directly multiplying W, the arithmetic unit 10 processes W using a pre-separated WC1 and W C2 The data for the two matrices is stored in memory 10b, etc., and these matrices are used in matrix multiplication operations. Here, W C1 In the following explanation, corresponds to the first separable matrix, W C2 This corresponds to the second separable matrix in the following explanation.

[0026] To perform multiplication of the first matrix and the second matrix, the dedicated circuit 20 includes a first arithmetic unit 30, a second arithmetic unit 40, and a matrix addition unit 50, as shown in Figure 2. The first arithmetic unit 30 performs the first separable matrix product, which is the matrix product of the first separable matrix and the second matrix. The second arithmetic unit 40 performs the second separable matrix product, which is the matrix product of the second separable matrix and the second matrix.

[0027] Then, the matrix addition unit 50 can obtain the matrix product of the first matrix and the second matrix by adding the first separable matrix product and the second separable matrix product. As shown in Equation 1, the matrix addition unit 50 may further add a bias matrix in the operation of adding the first separable matrix product and the second separable matrix product.

[0028] Here, the first arithmetic unit 30 and the second arithmetic unit 40 may perform calculations in different ways. These differences in methods may be due to differences in the hardware used for the calculations. They may also be due to differences in the format used to represent the first separable matrix and the format used to represent the second separable matrix. Furthermore, they may be due to differences in the characteristics of the first separable matrix and the characteristics of the second separable matrix.

[0029] As shown in Figure 3, the dedicated circuit 20 is configured to execute the calculation by the first calculation unit 30 (P1 in Figure 3) and the calculation by the second calculation unit 40 (P2 in Figure 3) in parallel in time. After both calculations are completed, the calculation by the matrix addition unit 50 (P3 in Figure 3) is executed. This reduces the overall processing time compared to executing the calculation by the first calculation unit 30 and the calculation by the second calculation unit 40 sequentially.

[0030] The process of separating the first matrix into a first separation matrix and a second separation matrix may be implemented by the CPU 10a executing a computer program in the arithmetic unit 10, or by a dedicated circuit 20, but is usually implemented outside the arithmetic unit 10. For example, the weights become all known parameters once the neural network model has been trained.

[0031] Therefore, before implementing the model on the mobile device, it is possible to perform training outside of the computing unit 10 and pre-separate the matrix (first matrix) containing the weights determined in the trained model into a first separation matrix and a second separation matrix. Then, by storing the first and second separation matrices in memory 10b or the like instead of the first matrix, the neural network model can be implemented on the mobile device.

[0032] Here, we will explain an example of separation of the first matrix. The first matrix is ​​separated based on the numerical distribution, which is the distribution of the numbers stored in the first matrix. If the first matrix is, for example, a matrix that stores the weights of a single convolutional layer, then the numerical distribution of the weights of that convolutional layer is statistically processed. The numerical distribution can be distinguished into a principal part, which corresponds to the principal part close to the representative value, and an outlier part, which corresponds to the outliers (the part that looks like so-called noise). The representative value may be the mean, the median, or the mode.

[0033] The two distinct parts are separated to generate a first separation matrix and a second separation matrix. The first separation matrix stores the numerical values ​​corresponding to the main part of the first matrix. In this separation example, the first separation matrix is ​​a dense matrix, meaning it has a relatively higher density of stored numerical values ​​than the second separation matrix. In the separation process, the first separation matrix may be approximated. In the approximation process, lower-order values ​​(e.g., specific numbers after the decimal point) may be rounded, rounded up, or rounded down. Specific examples of approximation are shown in the second embodiment and subsequent embodiments. Furthermore, approximation means that the matrix obtained by adding the separated first and second separation matrices does not necessarily match the first matrix.

[0034] After approximation, the first separable matrix may be compressed. Compression is, for example, quantization. Quantization may mean reducing the number of bits used to represent the numerical value from a high number of bits to a low number of bits. Here, "low bit" means, for example, 8 bits or less. The number of bits of the numerical value after quantization may be, for example, 8 bits or less, and more preferably 4 bits or less. Furthermore, the numerical value may be represented by 2 bits, 3 bits, etc. In quantization, the size (i.e., scale) of the numerical value may be adjusted by scaling calculation.

[0035] On the other hand, the second separable matrix stores the numerical values ​​corresponding to the outliers in the first matrix. In this separation example, the second separable matrix is ​​a sparse matrix, meaning it has lower sparsity than the first separable matrix. Sparsity may be defined as the number of zero elements in a matrix divided by the total number of elements. Sparsity may also be expressed as a percentage. In Japanese, sparsity is also called sparseness or sparse density. This second separable matrix may or may not be approximated during the separation process, and may or may not be compressed.

[0036] This sparse matrix may be an FP sparse matrix (Floating Point sparse matrix). An FP sparse matrix is a sparse matrix in which non-zero elements storing non-zero numerical values are represented by floating-point numbers. As described above, making the second separation matrix sparse is also one aspect of compression processing.

[0037] Furthermore, instead of being made sparse, the second separation matrix may be made into a low-rank matrix. This low-rank matrix formation is also one aspect of compression processing. In this example, the second separation matrix is a matrix having a lower rank than the first separation matrix.

[0038] X represented by mathematical formula 1 C , W C1 and W C2 as a combination of the above, the combinations shown in Table 1 below can be employed. This table also includes the above-described separation example (Pattern 1).

[0039]

Table 1

[0040] Pattern 2 is suitable for providing a GPU in the dedicated circuit 20 and executing QLoRA (Quantized Low-Rank Adaptation) at high speed on the GPU. QLoRA is a processing method that significantly reduces memory usage by using NF4 (4-bit NormalFloat).

[0041] In Pattern 3, W C1 X C enables the use of a low-bit multiplier in the operation of . In Pattern 4, W C2 is also reduced in bit width, whereby W C2X C This allows the use of low-bit multipliers in calculations. Pattern 4 is effective for GPU calculations, but is particularly effective for calculations in circuits (including ICs) and FPGAs (Field Programmable Gate Arrays).

[0042] According to the first embodiment described above, the first matrix is ​​separated into a first separation matrix and a second separation matrix, and the matrix product with the second matrix is ​​calculated in parallel in time. This suppresses the expansion of the calculation scale of a single matrix product and speeds up the calculation by processing in parallel in time. Furthermore, it is possible to optimize the calculation of the first separation matrix product and the calculation of the second separation matrix according to the separation pattern based on the numerical distribution of the first and second separation matrices, thereby suppressing a decrease in calculation accuracy. As a result, it is possible to process the calculation appropriately.

[0043] Furthermore, according to the first embodiment, at least one of the first and second separation matrices is approximated during separation, and the data size is compressed. Compression accompanied by approximation can improve the compression ratio while suppressing a decrease in calculation accuracy.

[0044] Furthermore, according to the first embodiment, the first separable matrix is ​​a matrix from which the main part of the numerical distribution of the first matrix has been separated. The second separable matrix is ​​a sparse matrix from which the outlier parts of the numerical distribution, other than the main part of the first matrix, have been separated, and which is more spars than the first separable matrix. In this separation configuration, the calculation method for the product of the first separable matrix and the calculation method for the product of the second separable matrix can be optimized according to their respective sparsity, thereby making the calculation process more efficient.

[0045] Furthermore, according to the first embodiment, the first separable matrix is ​​a matrix from which the main part of the numerical distribution of the first matrix has been separated. The second separable matrix is ​​a low-rank matrix from which the outlier parts of the numerical distribution, which are not the main part of the first matrix, have been separated, and which has a lower rank than the first separable matrix. In this separation configuration, the calculation method for the product of the first separable matrix and the calculation method for the product of the second separable matrix can be optimized according to their respective ranks, thereby making the calculation process more efficient.

[0046] (Second Embodiment) As shown in Figures 4 and 5, the second embodiment is a modified version of the first embodiment. The second embodiment will be described focusing on the differences from the first embodiment.

[0047] In the second embodiment, the quantization of the first embodiment will be described in detail. As the quantization method, symmetric quantization or asymmetric quantization may be employed.

[0048] As shown in Figure 4, symmetric quantization is a quantization method that maps numerical data to a symmetric range centered on 0 as the zero point. The distribution D1 of numbers stored in the first separable matrix before quantization is assumed to be, for example, asymmetric with respect to 0. Here, the minimum value of the numbers included in the distribution D1 is x min Let the maximum value of the numbers included in distribution D1 be x max (Note that in Figures 4 and 5, the subscript 'f' is placed to the lower right of each symbol.)

[0049] The quantized distribution DS consists of multiple regions with a width of Δx. For example, the quantized value is the median of each region. In symmetric quantization, the central region of each area has a median of 0. The regions at both ends of each area are x min ,x max The median is set based on the value with the largest absolute value among them. In Figure 4, the number with the largest absolute value is x max Therefore, the region located on the minimum side of each region is -xmax With the median being +x, the region located on the side of the maximum value within each region is +x max This is used as the median.

[0050] As shown in Figure 5, asymmetric quantization is a quantization method that maps numerical data to a range asymmetric with respect to 0. That is, in asymmetric quantization, the zero point in quantization can be 0. The distribution D1 is assumed to be the same as in Figure 4. Here, μ is the value representing the center of the distribution, and x min ,x max They are located at an equal distance from each other.

[0051] The quantized distribution DA, as in Figure 4, consists of multiple regions with a width of Δx, and for example, the quantized value is the median of each region. The median in the central region is the rounded value of μ after the rounding process is performed. Zx is the difference between the rounded value and μ. The region located on the minimum value side of each region is x min The value obtained by adding Zx to this value is taken as the median. The region located on the side of the minimum value within each region is x max The median is the value obtained by adding Zx to this value.

[0052] Thus, asymmetric quantization can suppress quantization errors even for unbalanced distributions D1. On the other hand, symmetric quantization allows for a multiplier that is one bit smaller than that used in asymmetric quantization.

[0053] Therefore, in the separation example shown in the first embodiment, when the first matrix is ​​separated into two, the numerical distribution of the main part, after the outlier portion has been excluded, tends to be a distribution that is symmetric with respect to 0. For this reason, even if symmetric quantization is applied in the separation to the first separation matrix, the reduction in quantization error is not significant. Thus, it is possible to achieve both suppression of quantization error and miniaturization of the multiplier.

[0054] The formula for this reasoning can be expressed as shown in Equation 2 below, provided that the equation shown in Equation 3 holds true.

[0055]

number

number

[0056] Furthermore, the presence of sparse matrices suppresses the increase in quantization error, even if the conditions shown in Equation 4 are met for the zero points of the weights.

[0057]

number

[0058] Furthermore, as another example, an embodiment corresponding to pattern 3 of the first embodiment will be described. In this embodiment, symmetric quantization is performed on sparse matrices in addition to dense matrices during separation. In this embodiment, the inference formulation corresponding to equation 2 above can be expressed as shown in equation 5 below.

[0059]

number

[0060]

number

[0061] According to the second embodiment described above, quantization is applied to the first separable matrix. In the first separable matrix, which tends to require a large amount of processing because the main part is stored there, the data size is compressed by quantization, so the computational processing can be made more efficient.

[0062] Furthermore, according to the second embodiment, quantization is also applied to the second separable matrix. Since the data size of the second separable matrix is ​​compressed by quantization in accordance with the first separable matrix, the calculation of the second separable matrix product is made more efficient and easier to perform in parallel with the calculation of the first separable matrix product.

[0063] Furthermore, according to the second embodiment, the quantization applied to the first separable matrix is ​​symmetric quantization. By applying symmetric quantization to the first separable matrix, which tends to have a numerical distribution symmetric with respect to 0 due to the separation of outlier portions into the second matrix, it is possible to reduce the amount of processing and improve the efficiency of computation while suppressing the increase in quantization error. Moreover, if the quantization applied to the second separable matrix is ​​also symmetric quantization, the amount of processing can be reduced even further and the efficiency of computation can be improved.

[0064] (Third embodiment) As shown in Figure 6, the third embodiment is a modified version of the first embodiment. The third embodiment will be described focusing on the differences from the first embodiment.

[0065] In the third embodiment, the dedicated circuit 320 combines a dense matrix core 21 and a sparse matrix core 25 to perform multiplication of the first matrix and the second matrix. Here, "core" refers to a hardware component. In particular, when the core is implemented by a semiconductor integrated circuit, it refers to an IP core (Intellectual Property Core). The dense matrix core 21 and the sparse matrix core 25 may be mounted on a substrate as separate semiconductor chips, or they may be packaged as a single semiconductor chip. Furthermore, the dense matrix core 21 and the sparse matrix core 25 may be incorporated as circuits inside a GPU or CPU.

[0066] The dense matrix core 21 is also called a Dense Core. The dense matrix core 21 is a core optimized to efficiently process dense matrix operations. The dense matrix core 21 stores all values, including zeros, in the matrix multiplication operation in memory and processes the operation. This memory may be a dedicated memory located inside the dense matrix core 21, a memory shared with the sparse matrix core 25, or even a memory 10b, RAM 10c, etc., located outside the dedicated circuit 320.

[0067] The sparse matrix core 25 is also called the Sparse Core. The sparse matrix core 25 is a core optimized to efficiently process sparse matrix operations. In matrix multiplication operations, the sparse matrix core 25 skips processing of zero values ​​and processes only non-zero values. This is because multiplication involving zero results in zero, and processing it with a normal multiplier would result in slow processing speed. In contrast to the dense matrix core 21, the sparse matrix core 25 stores only the non-zero values ​​among the numbers handled in matrix multiplication operations in memory and processes the operations. The memory here may be a dedicated memory provided inside the sparse matrix core 25, a memory provided for shared use with the dense matrix core 21, or even a memory 10b, RAM 10c, etc., provided outside the dedicated circuit 320.

[0068] In the following section, the processing in the neural network in the third embodiment will be specifically explained using mathematical formulas. In a multilayered convolutional network, the convolution operation of the L+1 layer following the L layer is represented by the following equation 7. Here, w is the weight, x is the feature, y is the result of the operation, and the superscript represents the layer number.

[0069]

number

[0070]

number

number

[0071]

number

[0072]

number

number

[0073] The dense matrix core 21 has a multiplier 22 and a storage unit 23, and corresponds to the first arithmetic unit 30 described in the first embodiment. The multiplier 22 performs the multiplication of the first term in parentheses in the third row of number 12, i.e., dense matrix multiplication. The storage unit 23 sequentially stores the results of the multiplication performed by the multiplier 22 until they are added later by the sparse matrix core 25.

[0074] The sparse matrix core 25 includes a multiplier 26, a matrix adder 27, a storage unit 28, and a scale calculation circuit 29, and corresponds to the second arithmetic unit 40 and matrix adder 50 described in the first embodiment. The sparse matrix core 25 performs calculations other than the multiplication of the dense matrix in the third row of equation 12 (in other words, subsequent calculations).

[0075] The multiplier 26 performs multiplication of the second term inside the parentheses in the third row of equation 12, i.e., sparse matrix multiplication. The matrix adder 27 performs multiplication of the first and second terms inside the parentheses in the third row of equation 12. The storage unit 28 sequentially stores the results of the multiplications performed by the matrix adder 27 until the final result is output.

[0076] The scale calculation circuit 29 is used when performing scale calculations associated with the operations of the multiplier 26 and the matrix adder 27. The scale calculation circuit 29 may include a multiplier 29a and a barrel shifter 29b. The multiplier 29a is, for example, a 32-bit multiplier and performs multiplication by m. The barrel shifter 29b is, for example, a 48-bit barrel shifter constructed by arranging multiplexers and performs multiplication by n related to powers of 2 by bit shifting. Based on equation 12, by consolidating and sharing the scale calculation on the sparse matrix core 25 side, the circuit of the dedicated circuit 320 can be reduced in size.

[0077] According to the third embodiment described above, quantization is applied to the first and second separation matrices. The arithmetic unit 10 is equipped with a scale calculation circuit 29 that adjusts the scale in the quantization of both matrices, which is shared between the first arithmetic unit 30 and the second arithmetic unit 40. Sharing suppresses an increase in circuit size and makes calculations more efficient.

[0078] Furthermore, according to the third embodiment, the scale calculation circuit 29 is incorporated into the sparse matrix core 25, which is hardware prone to intermittent operation, among the dense matrix core 21, which is hardware that realizes the functions of the first arithmetic unit 30, and the sparse matrix core 25, which is hardware that realizes the functions of the second arithmetic unit 40, separate from the dense matrix core 21. By having the shared scale calculation performed on hardware prone to intermittent operation in matrix multiplication operations, the processing load of the first arithmetic unit 30 and the second arithmetic unit 40 can be balanced, and the calculations can be easily executed in parallel in terms of time.

[0079] (Fourth Embodiment) As shown in the figure, the fourth embodiment is a modified version of the second embodiment. The fourth embodiment will be described focusing on the differences from the second embodiment.

[0080] In the fourth embodiment, the formulation shown in equation 5 of the second embodiment is further approximated as shown in equation 13 below.

[0081]

number

[0082]

number

[0083] According to the fourth embodiment described above, the scaling coefficient includes a power of 2. The scaling calculation circuit 29 includes a barrel shifter 29b that bit-shifts the numerical value to be quantized. By handling at least a portion of the scaling calculation with bit shifting, the circuit can be configured more simply than performing the operation with a normal multiplier.

[0084] Furthermore, according to the fourth embodiment, the exponent in the power of 2 is set to a range from -2 to 2. Since it is possible to configure the barrel shifter 29b based on this range limitation, it is possible to suppress an increase in circuit size.

[0085] (Fifth embodiment) As shown in Figures 7 and 8, the fifth embodiment is a modified version of the second to fourth embodiments. The fifth embodiment will be described focusing on the differences from the second to fourth embodiments.

[0086] In the fifth embodiment, the calculation process of the second arithmetic unit 40 is simplified. Specifically, in the calculation process CC1 between the first separable matrix as a dense matrix and the second matrix, all elements of the matrix are multiplied sequentially regardless of whether the numerical value is 0 or not.

[0087] In contrast, in the CC2 operation between the second separable matrix (as a sparse matrix) and the second matrix, mask patterns MP, MP1, and MP2 are generated that indicate where in the second separable matrix elements that contain non-zero values ​​(hereinafter referred to as non-zero elements) are located.

[0088] For example, in the mask pattern MP1 shown in Figure 7, elements that contain 0 in the second separable matrix will contain 0, and elements that contain a non-zero value in the second separable matrix will contain 1. This mask pattern MP1 may also be called a mask matrix.

[0089] In other words, even if a numerical value such as 2 is stored in the second separable matrix, 1 will be stored in the mask pattern MP1. That is, the mask pattern MP1 represents a sparse matrix using binary values ​​of 0 and 1. If the second separable matrix separates the weights that can be determined at the time of training, the mask pattern MP1 can be prepared in advance and stored in memory 10b, etc.

[0090] The second arithmetic unit 40 then refers to the mask pattern MP1, extracts non-zero elements in the second separable matrix, and performs multiplication only on the non-zero elements. The extraction assumes that the size of the first separable matrix and the second separable matrix are the same. Therefore, the second arithmetic unit 40 refers to the multiplication CC1, and at the same timing as the position in the first separable matrix where the multiplication CC1 is being performed, the second arithmetic unit 40 at the corresponding position in the second separable matrix determines whether the mask pattern MP1 is 0 or 1, thereby extracting elements that will be skipped in the arithmetic process CC2.

[0091] Furthermore, as shown in Figure 8, the location of non-zero elements in the mask pattern MP2 may be indicated not by two parameters, 0, 1 or row and column, for each element, but by a single parameter called length L1 and L2. This allows the mask pattern MP2 to be represented using only a number of lengths equal to the total number of non-zero elements, without having to represent all elements with binary values ​​of 0 and 1.

[0092] Lengths L1 and L2 indicate the length of the matrix elements along each row. When the end of a row is reached, the system jumps to the beginning of the next row, and the lengths of multiple rows are added together. Length L1, which indicates the position of the first non-zero element, represents the distance from the beginning of the first row of the matrix to the position of the first non-zero element. For example, if the first non-zero element is the 20th element from the beginning of the first row, then L1 = 20.

[0093] The length L2, which indicates the position of the next non-zero element, represents the distance from the previous non-zero element or the next element (whichever is predefined) to the position of the next non-zero element. For example, if the second non-zero element is the 220th element from the start of the first row, and the 200th element from the element following the first non-zero element, then L2 = 200.

[0094] According to the fifth embodiment described above, in the operation of multiplying a sparse matrix second separable matrix, by using a mask pattern MP1 in which elements of the second separable matrix whose stored numerical value is 0 are represented as 0 and non-zero elements are represented as 1, multiplication involving 0 (multiplication with a multiplier of 0) is skipped. Because the matrix elements to be skipped can be determined by a simple check due to the existence of the mask, the processing can be sped up.

[0095] Furthermore, according to the fifth embodiment, in the operation of multiplying a sparse matrix second separable matrix, a mask pattern MP2 is used, which represents the position of elements in the second separable matrix whose stored numerical values ​​are not zero with a single parameter. This allows for skipping multiplication involving zero (multiplication where the multiplier is zero). The presence of the mask makes it easy to identify the position of the matrix elements to be skipped, thus speeding up the processing.

[0096] (Sixth Embodiment) As shown in Figure 9, the sixth embodiment is a modified version of the third and fourth embodiments. The sixth embodiment will be described focusing on the differences from the third embodiment.

[0097] In the sparse matrix core 25, if the sparsity of the second separable matrix is ​​extremely high (for example, 98% or more), the sparse matrix core 25 is expected to operate intermittently. That is, the multiplier of the sparse matrix core 25 can spend more time on a single calculation. Therefore, in the sparse matrix core 25 of the sixth embodiment, the multiplier 26 is implemented using time-division multiplexing. This reduces the circuit size of the multiplier 26.

[0098] For example, if the multiplier 26 is 4 bits and the time division ratio is 2, then the 4-bit operation will be performed twice. As a result, the multiplier 26 can effectively perform an 8-bit operation.

[0099] The time division number should be set according to the sparsity of the second separable matrix and the processing performance of the dense matrix core 21. Specifically, as shown in Figure 9, one unit of multiplication in the multiplier 22 of the dense matrix core 21 is denoted as Px, and calculations are performed in the order of, for example, Px1, Px2.

[0100] On the other hand, in the multiplier 26 of the sparse matrix core 25, let S be the multiplication corresponding to Px, and the calculations are performed in the order S1, S2. Here, S1 and S2 are divided by a time division number N, for example, S1 is time-divided into N operations such as S11, S12, ..., S1N. For example, let Px1 and S1 be corresponding elements (the same elements separated in the first matrix). In this case, the time division number N should be set so that the total time taken for operations S11 to S1N is less than or equal to the time taken for the operation Px1.

[0101] Furthermore, if the second matrix is ​​quantized to m bits and the second separable matrix is ​​quantized to n bits, the time division ratio should be set to be less than or equal to the multiplier of m and n.

[0102] (Seventh Embodiment) As shown in Figures 10 and 11, the seventh embodiment is a modification of the first embodiment. The seventh embodiment will be described focusing on the differences from the first embodiment.

[0103] As shown in Figure 10, the computing device 90 of the seventh embodiment does not have a dedicated circuit specifically for neural network computation. In the computing device 90, the GPU 90a can execute all neural network computation according to the processing method defined in the computer program (arithmetic program) stored in memory 90b.

[0104] The GPU90a is a processing unit with numerous computing cores, capable of processing multiple operations in parallel over time. For example, the GPU90a operates in SIMD (Single Instruction Multiple Data) format.

[0105] The GPU90a reads the calculation program and executes the processing corresponding to the first calculation unit 30 and the processing corresponding to the second calculation unit 40 in the first embodiment in parallel in terms of time (S1 in Figure 11). Here, the processing corresponding to the first calculation unit 30 includes a first calculation that calculates the first separation matrix product, which is the matrix product of the first separation matrix and the second matrix. The processing corresponding to the second calculation unit 40 includes a second calculation that calculates the second separation matrix product, which is the matrix product of the second separation matrix and the second input matrix.

[0106] The GPU90a reads the calculation program, processes the first and second calculations in parallel, and then executes the process corresponding to the matrix addition unit 50 of the first embodiment (S2 in Figure 11). The process corresponding to the matrix addition unit 50 includes matrix addition, which adds the first separated matrix product and the second separated matrix product to obtain the matrix product of the first matrix and the second matrix.

[0107] (Other embodiments) Although several embodiments have been described above, this disclosure is not limited to those embodiments and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0108] In another embodiment, the arithmetic unit 10 may have a third arithmetic unit, a fourth arithmetic unit, etc., in addition to the first arithmetic unit 30 and the second arithmetic unit 40, so that three or more arithmetic units can perform calculations in parallel in time. In this case, the first matrix may be separated into three or more matrices. The matrix addition unit 50 may then add the three or more matrix products calculated by each arithmetic unit. Similarly, the arithmetic unit 90 may process three or more operations related to the matrix product calculation of the separated matrices obtained by separating the first matrix in parallel in time.

[0109] Furthermore, as another embodiment relating to the seventh embodiment, the arithmetic unit 90 may be provided with a processing unit other than a GPU that can process multiple processes in parallel in time, instead of the GPU 90a. For example, the processing unit may be a CPU equipped with multiple cores. The arithmetic program may be configured to have the processing unit execute the first operation, the second operation, and matrix addition.

[0110] In another embodiment, the arithmetic units 10 and 90 may be packaged in the form of a single semiconductor chip.

[0111] In another embodiment, the arithmetic units 10 and 90 may perform matrix multiplication operations that are not used in neural networks. For example, the matrix multiplication operation may be an operation used in image processing.

[0112] Furthermore, in other embodiments, the computing units 10 and 90 do not have to be mounted on a vehicle. The computing units 10 and 90 may be mounted on a mobile device such as an aircraft, drone, or ship. The computing units 10 and 90 may not be mounted on a mobile device and may be used, for example, in an edge terminal. The computing units 10 and 90 may be mounted on consumer products such as robotic vacuum cleaners.

[0113] (Disclosure of technical ideas) This specification discloses several technical concepts, as listed in the following paragraphs. Some paragraphs are written in a multiple dependent form, where subsequent paragraphs alternately refer to preceding paragraphs. These paragraphs written in a multiple dependent form define several technical concepts.

[0114] <Technical philosophy 1> A arithmetic unit that performs the matrix product of a first matrix and a second matrix, A first arithmetic unit (30) performs a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution which is the distribution of numerical values ​​stored in the first matrix. The first arithmetic unit is an arithmetic unit that can perform operations in parallel in time, and includes a second arithmetic unit (40) that performs a second separation matrix product, which is the matrix product of the second matrix and the second matrix, in which a second separation matrix is ​​obtained by separating numerical values ​​from the first separation matrix that are different from the numerical values ​​stored in the first separation matrix from the first matrix. An arithmetic device comprising: a matrix addition unit (50) that adds the first separation matrix product and the second separation matrix product to obtain the matrix product of the first matrix and the second matrix.

[0115] <Technical philosophy 2> The computing device according to technical idea 1, wherein at least one of the first separation matrix and the second separation matrix is ​​approximated in the separation process.

[0116] <Technical philosophy 3> The arithmetic device according to technical idea 2, wherein at least one of the above is subjected to data size compression processing.

[0117] <Technical philosophy 4> The first separation matrix is ​​a matrix from which the main part of the numerical distribution has been separated from the first matrix. The arithmetic device according to any one of technical ideas 1 to 3, wherein the second separable matrix is ​​a sparse matrix in which the outlier portion corresponding to the outliers of the numerical distribution, other than the main portion of the first matrix, is separated, and has higher sparsity than the first separable matrix.

[0118] <Technical philosophy 5> The first separation matrix is ​​a matrix from which the main part of the numerical distribution has been separated from the first matrix. The arithmetic device according to any one of technical ideas 1 to 3, wherein the second separable matrix is ​​a low-rank matrix in which the outlier portion corresponding to the outliers of the numerical distribution, other than the main portion of the first matrix, is separated, and the rank of the matrix is ​​lower than that of the first separable matrix.

[0119] <Technical philosophy 6> The computing device according to technical idea 4 or 5, wherein quantization is applied to the first separable matrix.

[0120] <Technical philosophy 7> The quantization is symmetric quantization, as described in Technical Idea 6 of the computing device.

[0121] <Technical philosophy 8> The computing device according to technical idea 6 or 7, wherein quantization is applied to the second separable matrix.

[0122] <Technical philosophy 9> The aforementioned quantization is symmetric quantization, as described in Technical Idea 8 of the computing device.

[0123] <Technical Thought 10> Quantization is applied to the first and second separation matrices. The arithmetic device according to any one of technical ideas 1 to 9, wherein a scale calculation circuit (29) for adjusting the scale in the quantization of both matrices is shared between the first arithmetic unit and the second arithmetic unit.

[0124] <Technical Thought 11> The arithmetic device according to technical concept 10, wherein the scale calculation circuit is incorporated into the hardware (25) which is prone to intermittent operation, among the hardware (21) that realizes the function of the first arithmetic unit and the hardware (25) that realizes the function of the second arithmetic unit, which is separate from this hardware.

[0125] <Technical Thought 12> Quantization is applied to the first and second separation matrices. The quantization includes a scale calculation circuit (29) for adjusting the scale, The coefficients relating to the scale include powers of 2, The scale calculation circuit is an arithmetic device according to technical idea 1, which includes a barrel shifter (29b) for bit-shifting the numerical value to be quantized.

[0126] <Technical Thought 13> The arithmetic device according to technical idea 12, wherein the exponent in the power of 2 is set in the range of -2 to 2.

[0127] <Technical Thought 14> The first arithmetic unit performs multiplication of all matrix elements of the first separable matrix, The second arithmetic unit refers to the multiplication performed by the first arithmetic unit and uses a mask (MP1) that represents elements of the second separable matrix whose stored numerical value is 0 as 0 and elements that are not 0 as 1 to determine whether the matrix element of the second separable matrix corresponding to the matrix element for which multiplication is being performed in the first arithmetic unit is 0, and skips the multiplication involving 0, as described in technical concept 4.

[0128] <Technical Thought 15> The second arithmetic unit is an arithmetic device according to technical concept 4, which skips multiplication involving zero by using a mask (MP2) in which the positions of elements in the second separable matrix whose stored numerical values ​​are not zero are represented by a single parameter.

[0129] <Technical Thought 16> The second arithmetic unit is an arithmetic device according to any one of technical ideas 1 to 15, which performs the multiplication of the second separable matrix product using time-division processing.

[0130] <Technical Thought 17> The first matrix mentioned above is a matrix representing the weights of the neural network, The second matrix is ​​a matrix representing the features of the neural network, as described in any one of the technical ideas 1 to 16. [Explanation of Symbols]

[0131] 10,90: Arithmetic unit, 30: First arithmetic unit, 40: Second arithmetic unit, 50: Matrix addition unit, 90a: GPU (processing unit)

Claims

1. A arithmetic unit that performs the matrix product of a first matrix and a second matrix, A first arithmetic unit (30) performs a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution which is the distribution of numerical values ​​stored in the first matrix. The first arithmetic unit is an arithmetic unit that can perform operations in parallel in time, and includes a second arithmetic unit (40) that performs a second separation matrix product, which is the matrix product of the second matrix and the second matrix, in which a second separation matrix is ​​obtained by separating numerical values ​​from the first separation matrix that are different from the numerical values ​​stored in the first separation matrix from the first matrix. An arithmetic device comprising: a matrix addition unit (50) that adds the first separation matrix product and the second separation matrix product to obtain the matrix product of the first matrix and the second matrix.

2. The arithmetic device according to claim 1, wherein at least one of the first separation matrix and the second separation matrix is ​​approximated in the separation process.

3. The arithmetic device according to claim 2, wherein at least one of the above is subjected to data size compression processing.

4. The first separation matrix is ​​a matrix from which the main part of the numerical distribution has been separated from the first matrix. The arithmetic device according to claim 1, wherein the second separable matrix is ​​a sparse matrix that is more spars than the first separable matrix, in which the outlier portion of the first matrix, other than the main portion, corresponding to the outliers of the numerical distribution, is separated.

5. The first separation matrix is ​​a matrix from which the main part of the numerical distribution has been separated from the first matrix. The arithmetic device according to claim 1, wherein the second separable matrix is ​​a low-rank matrix in which the outlier portion of the first matrix, excluding the main portion and corresponding to the outliers of the numerical distribution, is separated, and the rank of the matrix is ​​lower than that of the first separable matrix.

6. The computing device according to claim 4 or 5, wherein quantization is applied to the first separable matrix.

7. The arithmetic device according to claim 6, wherein the quantization is symmetric quantization.

8. The arithmetic device according to claim 6, wherein quantization is applied to the second separable matrix.

9. The arithmetic apparatus according to claim 8, wherein the quantization is symmetric quantization.

10. Quantization is applied to the first and second separation matrices. The arithmetic device according to claim 1, further comprising a scale calculation circuit (29) for adjusting the scale in the quantization of both matrices, which is shared between the first arithmetic unit and the second arithmetic unit.

11. The arithmetic device according to claim 10, wherein the scale calculation circuit is incorporated into the hardware (25) which is prone to intermittent operation, among the hardware (21) which realizes the function of the first arithmetic unit and the hardware (25) which realizes the function of the second arithmetic unit, separate from this hardware.

12. Quantization is applied to the first and second separation matrices. The quantization includes a scale calculation circuit (29) for adjusting the scale, The coefficients relating to the scale include powers of 2, The arithmetic device according to claim 1, wherein the scale calculation circuit includes a barrel shifter (29b) for bit-shifting the numerical value to be quantized.

13. The arithmetic device according to claim 12, wherein the exponent in the power of 2 is set in the range of -2 to 2.

14. The first arithmetic unit performs multiplication of all matrix elements of the first separable matrix, The arithmetic device according to claim 4, wherein the second arithmetic unit refers to the multiplication performed by the first arithmetic unit and uses a mask (MP1) in which elements of the second separable matrix whose stored numerical value is 0 are represented as 0 and elements that are not 0 are represented as 1, thereby determining whether the matrix element of the second separable matrix corresponding to the matrix element for which multiplication is performed in the first arithmetic unit is 0, and skips the multiplication involving 0.

15. The arithmetic device according to claim 4, wherein the second arithmetic unit processes by skipping multiplications involving zero by using a mask (MP2) in which the positions of elements in the second separable matrix whose stored numerical values ​​are not zero are represented by a single parameter.

16. The arithmetic device according to claim 1, wherein the second arithmetic unit performs the multiplication of the second separable matrix product using time-division processing.

17. The first matrix mentioned above is a matrix representing the weights of the neural network, The computing device according to claim 1, wherein the second matrix is ​​a matrix representing the features of the neural network.

18. A method for performing the matrix product of a first matrix and a second matrix, A first operation is performed to perform a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution which is the distribution of numerical values ​​stored in the first matrix. The first operation is performed in parallel with the second separation matrix product, which is the matrix product of the second separation matrix and the second matrix, where the second separation matrix contains the numbers other than those stored in the first separation matrix, and the first separation matrix is ​​performed in parallel with the first operation. A method of operation that includes matrix addition, which involves adding the product of the first separable matrix and the product of the second separable matrix to obtain the matrix product of the first matrix and the second matrix.

19. A calculation program for causing a processing unit (90a) capable of processing multiple processes in parallel in time to perform the matrix product of a first matrix and a second matrix, In the aforementioned processing unit, A first operation is performed to perform a first separation matrix product, which is the matrix product of the first matrix and the second matrix, where a portion of the first matrix is ​​separated based on a numerical distribution which is the distribution of numerical values ​​stored in the first matrix. The first operation is performed in parallel in time, and is a second operation which is the matrix product of the second matrix, which is the matrix product of the second matrix and the second matrix, where the second matrix contains the numbers other than those stored in the first matrix. A program that performs matrix addition, which involves adding the product of the first and second separable matrices to obtain the matrix product of the first and second matrices.

Images