Matrix multiplication computing device, acceleration device, computing system, and related methods

Abstract

This application provides a matrix multiplication calculation unit, acceleration unit, computing system, and related methods. This solution is applicable to various chips, including those based on ARM and RISC-V instruction set architectures, such as IoT chips and audio / video chips. The matrix multiplication calculation unit includes: a control subunit, which operates in either a dense or sparse operation mode. In dense operation mode, it selects data from a first matrix as the first input data and data from a non-sparsed second matrix as the second input data and sends them to the operation subunit. In sparse operation mode, it selects data from a third matrix as the first input data and data from a sparsed fourth matrix as the second input data and sends them to the operation subunit. The operation subunit performs matrix multiplication operations based on the received first and second input data and outputs the results. This solution improves the applicability of the matrix multiplication calculation unit.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the chip technical field, and in particular to a matrix multiplication computing device, an acceleration device, a computing system and related methods. BACKGROUND

[0002] Sparse operation of a neural network can effectively reduce the bandwidth and storage of data and improve the energy efficiency of calculation. Compared with unstructured sparsity of a neural network, structured sparsity of a neural network can more effectively reduce the complexity of data fetching logic and calculation logic in circuit design, but not all neural networks are suitable for sparse operation because sparse operation reduces the accuracy of a neural network.

[0003] At present, a dedicated deep learning processor is developed and designed based on a neural network to be run.

[0004] However, in some application scenarios, the dedicated deep learning processor needs to run different neural networks, part of which has the need for sparse operation, and the other part of which does not have the need for sparse operation. However, the calculation unit in the deep learning processor is developed and designed for a specific application network, so that the calculation unit cannot simultaneously meet the operation logic of structured sparse neural networks and dense neural networks, resulting in poor applicability of the calculation unit. SUMMARY

[0005] Therefore, embodiments of the present application provide a matrix multiplication computing unit, an acceleration unit, a computing system and related methods to at least solve or alleviate the above problems.

[0006] According to a first aspect of embodiments of the present application, a matrix multiplication computing unit is provided, comprising: a control subunit and an operation subunit; the control subunit is configured to run in a dense operation mode or a sparse operation mode according to a received control signal, wherein in the dense operation mode, data in a first matrix is selected as first input data and sent to the operation subunit, and data in a non-sparse second matrix is selected as second input data and sent to the operation subunit, and in the sparse operation mode, data in a third matrix is selected as first input data and sent to the operation subunit, and data in a sparse fourth matrix is selected as second input data and sent to the operation subunit; the operation subunit is configured to perform matrix multiplication operation according to the received first input data and second input data, and output operation result.

[0007] According to a second aspect of embodiments of the present application, an acceleration unit is provided, comprising: the matrix multiplication computing unit of any one of the first aspect; a main control unit configured to control the operation of the computing unit.

[0008] According to a third aspect of the embodiments of the present application, a neural network accelerator is provided, comprising: the acceleration unit of any one of the second aspect; a scheduling unit configured to schedule the acceleration unit to obtain a first matrix and a non-sparse second matrix, and schedule the acceleration unit to perform a matrix multiplication operation on the first matrix and the second matrix, or schedule the acceleration unit to obtain a third matrix and a sparse fourth matrix, and schedule the acceleration unit to perform a matrix multiplication operation on the third matrix and the fourth matrix.

[0009] According to a fourth aspect of the embodiments of the present application, a computing system is provided, comprising: a processor, a direct memory access controller, a communication interface, a static random access memory, a communication bus, and the neural network accelerator of the third aspect; the processor, the direct memory access controller, the communication interface, and the static random access memory complete communication with each other through the communication bus; the processor, the direct memory access controller, the communication interface, and the neural network accelerator complete communication with each other through the communication bus; the static random access memory and the neural network accelerator exchange data through a private interface; the processor is configured to control the computing system and initialize the neural network accelerator.

[0010] According to a fifth aspect of the embodiments of the present application, a matrix multiplication calculation method is provided, comprising: receiving a control signal; determining a dense operation mode or a sparse operation mode according to the control signal, in the dense operation mode, selecting data in a first matrix as first input data and selecting data in a non-sparse second matrix as second input data, in the sparse operation mode, selecting data in a third matrix as first input data and selecting data in a sparse fourth matrix as second input data; performing a matrix multiplication operation according to the first input data and the second input data, and outputting an operation result.

[0011] According to the matrix method scheme provided in the embodiments of the present application, the control subunit can switch to the dense operation mode or the sparse operation mode according to the control signal, in the dense operation mode, data in the first matrix can be selected as the first input data and sent to the operation subunit, and data in the second matrix which is not sparse can be selected as the second input data and sent to the operation subunit, and the operation subunit calculates the product of the first matrix and the second matrix according to the received first input data and second input data, in the sparse operation mode, data in the third matrix can be selected as the first input data and sent to the operation subunit, and data in the fourth matrix which is sparse can be selected as the second input data and sent to the operation subunit, and the operation subunit calculates the product of the third matrix and the fourth matrix according to the received first input data and second input data. It can be seen that the control subunit can use different data selection logic to select data from the matrix in the dense operation mode and the sparse operation mode, and send the selected data to the operation subunit for multiplication and accumulation operation, so as to realize the matrix multiplication operation of the sparse matrix and the matrix multiplication operation of the non-sparse matrix, thereby meeting the operation logic of the structured sparse neural network and the dense neural network at the same time, so that the matrix multiplication calculation unit has strong applicability. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0013] Figure 1 is a schematic diagram of a computing system of an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a neural network accelerator of an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of an acceleration unit of an embodiment of the present application;

[0016] Figure 4 is an internal structure diagram of a matrix multiplication calculation unit of an embodiment of the present application;

[0017] Figure 5 is an internal structure diagram of a matrix multiplication calculation unit of another embodiment of the present application;

[0018] Figure 6 is an internal structure diagram of a matrix multiplication calculation unit of another embodiment of the present application;

[0019] Figure 7is a schematic diagram of the first matrix and the second matrix of one embodiment of the present application;

[0020] Figure 8 is a schematic diagram of matrix sparsification of one embodiment of the present application;

[0021] Figure 9 is a schematic diagram of matrix sparsification of another embodiment of the present application;

[0022] Figure 10 is a flow chart of a matrix multiplication calculation method of one embodiment of the present application. DETAILED DESCRIPTION

[0023] The present application is described below based on embodiments, but the present application is not limited to only these embodiments. In the following detailed description of the present application, some specific details are described in detail. The present application can also be fully understood without the description of these details by those skilled in the art. In order to avoid confusion of the essence of the present application, the well-known methods, processes, and flows are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0024] First, some nouns or terms appearing in the description of the embodiments of the present application are applicable to the following explanations.

[0025] Matrix multiplication: Let matrix A be an m x p matrix, matrix B be a p x n matrix, and matrix C be an m x n matrix, then matrix C is called the product of matrix A and matrix B, denoted as C = AB, where the element in the i-th row and the j-th column of matrix C can be expressed as

[0026] Matrix sparsification: Sparsify the matrix according to a predetermined rule, sparsify some elements in the original matrix to zero, and then remove the elements that are zero, and construct a new matrix from the non-zero elements. Matrix multiplication is performed on the sparsified matrix, which reduces the number of elements in the matrix, thereby reducing the data bandwidth and storage, and improving the computing efficiency.

[0027] Unstructured sparsification: Matrix sparsification is divided into unstructured sparsification and structured sparsification. Unstructured sparsification refers to sparsifying elements at random positions in the matrix to zero, and then removing the elements that are zero to construct a sparsified matrix from the non-zero elements.

[0028] Structured sparsification: Structured sparsification refers to sparsifying some elements in each group to zero, and then removing the elements that are zero to construct a sparsified matrix from the non-zero elements. For example, each row of the matrix is a data, and 4 or 8 consecutive rows are treated as a data group for sparsification. After sparsification of a data group, it becomes a matrix including 1 or 2 rows of elements, while the number of columns of the matrix remains unchanged.

[0029] Elements of a matrix: The elements of a matrix are the basic units that make up the matrix. The number of elements in a matrix is ​​equal to the product of the number of rows and columns. For example, if matrix A has m rows and p columns, then matrix A contains m × p elements.

[0030] Operation cycle: The operation cycle refers to the period at which a number is taken from the matrix in matrix multiplication, that is, how often a number is taken from the matrix. The operation cycle can be one clock cycle or multiple consecutive clock cycles.

[0031] Multiplier: A multiplier is an electronic device that performs the multiplication of two uncorrelated analog signals. It can also multiply two binary numbers. The multiplier and multiplicand are the inputs to the multiplier, and the product is the output, which is equal to the product of the multiplier and multiplicand. For example, if the inputs to the multiplier are X1 and X2, the output will be X1 × X2.

[0032] Accumulator: An accumulator is a component that produces the sum of two or more numbers. When multiple numbers are input into the accumulator sequentially, the accumulator calculates the sum of the input numbers. The composite number is the output of the accumulator, and it is equal to the sum of all the numbers input into the accumulator. For example, if the inputs to the accumulator are X1, X2, and X3, then the output of the accumulator is X1 + X2 + X3.

[0033] Acceleration Unit: A processing unit designed to improve the data processing speed in specialized fields (e.g., image processing, speech recognition, matrix multiplication calculation, etc.) where traditional processing units are inefficient. In this application embodiment, it is mainly a specialized processing unit designed to accelerate matrix multiplication operations.

[0034] Computing system

[0035] Figure 1 A schematic block diagram of a computing system 100 is shown. (e.g.) Figure 1 As shown, the computing system 100 can be built based on various types of processing units. The computing system 100 can be, for example, an Internet of Things (IoT) processing unit, a system-on-a-chip (SoC), a desktop processing system, or a data center processing system.

[0036] like Figure 1As shown, the computing system 100 includes a processor 10, a direct memory access controller 20, a communication interface 30, a static random access memory (SRAM) 40, a communication bus 50, and a neural network accelerator 60. The processor 10, SRAM 20, communication interface 30, and SRAM 40 can communicate with each other via the communication bus 50, and the processor 10, SRAM 20, communication interface 30, and neural network accelerator 60 can also communicate with each other via the communication bus 50. The SRAM 40 and neural network accelerator 60 exchange data through a proprietary interface.

[0037] The neural network accelerator 60 is used to accelerate algorithms in neural networks, mainly matrix multiplication, thereby improving the overall system's computational performance and energy efficiency.

[0038] The processor 10 is used for controlling the computing system 100 and for initializing and configuring the neural network accelerator 60.

[0039] The Direct Memory Access Controller (DMAC) 20 can move data from Double Data Rate (DDR) or Flash EEPROM Memory to Static Random Access Memory (SRAM) 40.

[0040] The neural network accelerator 60 can read data from the static random access memory 40 through a private interface and perform operations on the read data. For example, it can read a non-sparse matrix from the static random access memory 40 and perform matrix multiplication, or it can perform matrix multiplication from a sparse matrix in the static random access memory 40.

[0041] The communication interface 30 can connect to input / output devices such as storage devices, display devices, audio devices, keyboards, and mice. Storage devices can be hard disks, optical disks, flash memory, or other devices used for information storage, coupled to the communication bus 50 via the communication interface 30. Display devices can be coupled to the communication bus 50 via a corresponding graphics card for displaying information based on the display signals provided by the communication bus 50.

[0042] In this embodiment, the neural network accelerator 60 is a dedicated processing unit designed to accelerate matrix multiplication operations. It can be a processing unit specifically designed for matrix multiplication, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The processor 10 is the unit that schedules the neural network accelerator 60 and allocates the sequence of instructions to be executed to it. It can take various forms, such as a processing unit (CPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0043] The neural network accelerator 60 can perform matrix multiplication operations based on computer operation instructions. RISC-V is an open-source instruction set architecture based on the Reduced Instruction Set Computing (RISC) principle. It can be applied to various aspects of microcontrollers and FPGA chips, specifically in areas such as IoT security, industrial control, mobile phones, and personal computers. Because its design considers the realities of small size, speed, and low power consumption, it is particularly suitable for modern computing devices such as warehouse-scale cloud computers, high-end mobile phones, and tiny embedded systems. With the rise of AIoT (Artificial Intelligence of Things), the RISC-V instruction set architecture has received increasing attention and support and is expected to become the next generation of widely used CPU architecture.

[0044] The computer operation instructions in this application embodiment can be computer operation instructions based on the RISC-V instruction set architecture. Correspondingly, the neural network accelerator 60 can be designed based on the RISC-V instruction set. Specifically, the computing system provided in this application embodiment can be a chip designed using the RISC-V instruction set. This chip can execute executable code based on the configured instructions, thereby realizing matrix multiplication operations.

[0045] Traditional computing system architectures often allocate significant space to control and storage units, leaving insufficient space for computational units. While efficient for logic control, this approach is inefficient for large-scale parallel computing. Consequently, various specialized accelerators have been developed to significantly improve computational speed for different functions and domains. The neural network accelerator proposed in this application is a processing unit specifically designed to accelerate matrix multiplication operations in neural networks. It utilizes a structure combining control and data pathways to perform matrix multiplication operations on both sparse and non-sparse matrices.

[0046] The neural network accelerator proposed in this application, specifically designed for accelerating matrix multiplication, requires multiplying corresponding elements in the two matrices and then summing them when calculating the product of two matrices. The control path can operate in both dense and sparse operation modes. In these modes, the control path selects data from the two matrices according to different logics and sends it to the data path. The data path calculates the product of the data in the two matrices using multipliers and accumulators and then accumulates the results to obtain the multiplication result. Because the control path can extract data from the matrices according to different data extraction logics, it can extract data from both sparse and non-sparse matrices to perform matrix multiplication, thus making it applicable to both structured sparse neural networks and dense neural networks, and therefore possessing strong applicability.

[0047] Neural network accelerator

[0048] Figure 2 This is an internal structural diagram of a neural network accelerator according to an embodiment of this application. Figure 2 As shown, the neural network accelerator 60 includes an acceleration unit 61 and a scheduling unit 62.

[0049] Acceleration unit 61 is subject to scheduling by scheduling unit 62. Acceleration unit 61 is used to read and calculate data, write back the calculation results, and generate a signal indicating that the calculation of acceleration unit 61 is complete or an exception has occurred to scheduling unit 62. Scheduling unit 62 can trigger direct memory access controller 20 to move the calculated data and calculation-related commands to static random access memory 40, read the calculation-related commands from static random access memory 40, parse the read commands and send them to acceleration unit 61, and generate a calculation interruption or exception for neural network accelerator 60 after scheduling is completed.

[0050] Acceleration unit 61 can read two matrices for matrix multiplication from static random access memory 40, temporarily store the two matrices in an internal buffer, and then perform matrix multiplication on the read two matrices. The matrices read by acceleration unit 61 from static random access memory 40 can both be non-sparse matrices, or one non-sparse matrix and one sparse matrix, or two sparse matrices. Therefore, acceleration unit 61 can be applied to structured sparse neural networks and dense neural networks, thus having strong applicability.

[0051] Acceleration unit

[0052] Figure 3 This is an internal structural diagram of an acceleration unit according to an embodiment of this application. Figure 3As shown, the acceleration unit 61 includes a matrix multiplication calculation unit 611 and a main control unit 612. The main control unit 612 can generate and send control command streams within the acceleration unit 61 based on the configuration register group information, and report the completion of calculations and the status of any exceptions. Under the scheduling of the main control unit 612, the matrix multiplication calculation unit 611 can handle multiplication-accumulation, accumulation, and multiplication operations.

[0053] like Figure 3 As shown, the acceleration unit 61 also includes an address generation and bus access unit 613, a configuration register group 614, a data buffer unit 615, and a vector processing unit 616. After obtaining command stream information, the address generation and bus access unit 613 can generate corresponding read / write addresses, which are then converted into bus read / write operations, and the read data is written back to the data buffer unit 615. The configuration register group 614 stores the configuration information of the acceleration unit 61. The vector processing unit 616 is used for post-processing of matrix operation results.

[0054] The data caching unit 615 includes a first buffer, a second buffer, and a third buffer. The address generation and bus access unit 613, based on the control command stream generated by the main control unit 612, reads two matrices to be multiplied from the bus and stores them in the first and second buffers respectively. The matrix multiplication calculation unit 611 reads matrix data from the first and second buffers, performs matrix multiplication, and caches the result in the third buffer. The address generation and bus access unit 613 can read the result from the matrix multiplication calculation unit 611 from the third buffer and output the result via the bus.

[0055] When calculating the matrix multiplication of the first matrix and the non-sparsed second matrix, the first buffer is used to buffer the first matrix, the second buffer is used to buffer the second matrix, and the third buffer is used to buffer the result of the matrix multiplication of the first and second matrices. When calculating the matrix multiplication of the third matrix and the sparsed fourth matrix, the first buffer is used to buffer the third matrix, the second buffer is used to buffer the fourth matrix, and the third buffer is used to buffer the result of the matrix multiplication of the third and fourth matrices.

[0056] The matrix multiplication calculation unit 611 can determine whether the matrix to be multiplied is a sparse matrix and the sparsity ratio of the sparse matrix based on the control command stream generated by the main control unit 612. Then, the matrix multiplication calculation unit 611 can take data from the first buffer and the second buffer according to the corresponding data retrieval logic based on the control command stream to realize the matrix multiplication operation of sparse matrix and the matrix multiplication operation of non-sparse matrix.

[0057] The first, second, and third buffers can all be First-In-First-Out (FIFO) buffers. Since matrix multiplication requires multiplying elements in two matrices based on their row and column positions, using a FIFO buffer to store the matrix elements allows the matrix multiplication calculation unit 611 to sequentially select elements for multiplication and accumulation, ensuring the accuracy of the calculation result. Furthermore, the matrix multiplication calculation unit 611 stores the calculation result in the FIFO buffer, and the address generation and bus access unit 613 reads the calculation result from the FIFO buffer, ensuring the accuracy of the calculation result read by the address generation and bus access unit 613.

[0058] This application focuses on the matrix multiplication operation process performed by the matrix multiplication calculation unit 611. The matrix multiplication operation process will be described in detail later.

[0059] Matrix multiplication computation unit

[0060] Figure 4 This is an internal structural diagram of a matrix multiplication calculation unit according to an embodiment of this application. Figure 4 As shown, the matrix multiplication calculation unit 611 includes a control subunit 410 and an operation subunit 420. The control subunit 410 can operate in either a dense operation mode or a sparse operation mode based on received control signals. In dense operation mode, the control subunit 410 can select data from a first matrix as first input data and send it to the operation subunit 420, and select data from a non-sparsed second matrix as second input data and send it to the operation subunit 420. In sparse operation mode, the control subunit 410 can select data from a third matrix as first input data and send it to the operation subunit 420, and select data from a sparsed fourth matrix as second input data and send it to the operation subunit 420. The operation subunit 420 can perform matrix multiplication operations based on the received first and second input data and output the calculation result.

[0061] The operation subunit 420 is used to calculate the product of the first matrix and the second matrix, or the product of the third matrix and the fourth matrix. The control subunit 410 can select data from the first matrix or the third matrix as the first input data. The first matrix and the third matrix can be matrices or vectors. In matrix multiplication in neural networks, the data matrix is ​​usually multiplied by the weight matrix. The data matrix is ​​input into the neural network in real time, so the data matrix is ​​usually a non-sparse matrix, that is, the first matrix and the third matrix are non-sparse matrices.

[0062] In this embodiment, the control subunit 410 can switch to a dense operation mode or a sparse operation mode according to a control signal. In the dense operation mode, data can be selected from the first matrix as the first input data and sent to the operation subunit 420, and data from the non-sparsed second matrix can be selected as the second input data and sent to the operation subunit 420. The operation subunit 420 calculates the product of the first matrix and the second matrix based on the received first and second input data. In the sparse operation mode, data can be selected from the third matrix as the first input data and sent to the operation subunit 420, and data from the sparsed fourth matrix can be selected as the second input data and sent to the operation subunit 420. The operation subunit 420 calculates the product of the third matrix and the fourth matrix based on the received first and second input data. As can be seen, the control subunit 410 can use different data retrieval logics to retrieve data from the matrix in dense operation mode and sparse operation mode, and send the removed data to the operation subunit 420 for multiplication and accumulation operation, thereby realizing matrix multiplication operation of sparse matrix and matrix multiplication operation of non-sparse matrix. Thus, it can simultaneously satisfy the operation logic of structured sparse neural network and dense neural network, making the matrix multiplication calculation unit 611 highly applicable.

[0063] Figure 5 This is an internal structural diagram of the matrix multiplication calculation unit according to another embodiment of this application. Figure 5 As shown, the operation subunit 420 includes m multipliers 421 and m accumulators 422, where m is a positive integer greater than or equal to 2. Each multiplier 421 is connected to one accumulator 422, and different multipliers 421 are connected to different accumulators 422. The multiplier 421 can receive first input data and second input data sent by the control subunit 410, calculate the product of the received first input data and second input data, and send the calculated product to the connected accumulator 422. The accumulator 422 can sum the received products.

[0064] The number of multipliers 421 determines the number of matrix elements that can be processed per clock cycle during matrix multiplication. For example, m can take values ​​of 16, 32, or 64. When the number of multipliers 421 is 32, the control subunit 410 can send 32 first input data and 32 second input data to the operation subunit 420 per clock cycle. The operation subunit 420 can calculate the product of 32 pairs of matrix elements per clock cycle. It should be understood that a first input data refers to an element in the first or third matrix, and a second input data refers to an element in the second or fourth matrix.

[0065] In this embodiment, since matrix multiplication requires calculating the product of corresponding elements in two matrices and the sum of multiple products corresponding to elements in the same row or column, the operation subunit 420 includes multiple multipliers 421 and multiple accumulators 422. The multipliers 421 are used to calculate the product of the first input data and the second input data, and the accumulators 422 are used to accumulate and sum the products calculated by the connected multipliers 421. Therefore, the multipliers 421 and accumulators 422 provide the basic operation form required for matrix multiplication. After the control subunit 410 takes data from the two matrices as the first input data and the second input data according to the corresponding data retrieval logic, matrix multiplication of non-sparse matrices and matrix multiplication of sparse matrices can be realized, ensuring the applicability of the matrix multiplication calculation unit 611 while ensuring the accuracy of the matrix multiplication result.

[0066] Figure 6 This is a schematic diagram of a matrix multiplication calculation unit according to another embodiment of this application. Figure 6 As shown, the control subunit 410 includes a first selector 411, m second selectors 412, and a third selector 413. The first selector 411 is connected to each of the second selectors 412, each second selector 412 is connected to a multiplier 421, and different second selectors 412 are connected to different multipliers 421. The third selector 413 is connected to each of the multipliers 421.

[0067] In each operation cycle, the first selector 411 can sequentially select an element from the first matrix and send it to each of the second selectors 412. The second selector 412 can, in dense operation mode, send the element sent by the first selector 411 as first input data to the connected multiplier 421. The third selector 413 can, in each operation cycle, select m elements from the second matrix corresponding to the operation selected by the first selector 411 as second input data, and sequentially distribute the selected m second input data to the m multipliers 421.

[0068] In dense operation mode, matrix multiplication calculation unit 611 is used to calculate the product of the first matrix and the second matrix. Operation subunit 420 needs to calculate the product of each row element in the first matrix and each column element in the second matrix. Therefore, it is necessary to calculate the product of each element in the first matrix and each element in the corresponding row of the second matrix. For example, the first element of each row in the first matrix needs to be multiplied by each element in the first row of the second matrix. For this purpose, the first selector 411 can select an element from the first matrix in each operation cycle and send it to each second selector 412. At the same time, the third selector 413 selects q elements from the row of the second matrix whose row number is equal to the column number of the element selected by the first selector 411, and sends the selected q elements to q multipliers 421, where q is a positive integer less than or equal to m.

[0069] It should be noted that in dense operation mode, the second selector 412 can select the element from the first selector 411 as the first input data according to the received control signal.

[0070] For ease of explanation, the following explanation uses the first matrix as a row vector. In practical applications, the first matrix can be a row vector or a matrix containing multiple rows. Since the processing for each row is the same, this application only uses the first distance as a row vector as an example.

[0071] Figure 7 This is a schematic diagram of the first matrix and the second matrix according to an embodiment of this application. Figure 7 As shown, the first matrix A1 contains 32 elements, and the second matrix B1 is a 32×64 matrix. When m equals 32, the first selector 411 selects a1 from the first matrix A1 as the first input data and sends it to each of the second selectors 412. The second selectors 412 send a1 to the connected multiplier 421. The third selector 413 selects b from the second matrix B1. 1,1 To b 1,32 As the second input data, and b 1,i Send it to the i-th multiplier 421, for example, send b 1,1 Send it to the first multiplier 421, and b 1,2 Send it to the second multiplier 421, and b 1,32 The i-th multiplier 421 is sent to the 32nd multiplier 421, which calculates a1×b. 1,i Then, a1×b 1,i The data is sent to the i-th accumulator 422 connected to it. The first selector 411 selects a2 from the first matrix A1 as the first input data and sends it to each of the second selectors 412. The second selectors 412 send a2 to the connected multiplier 421. The third selector 413 selects b from the second matrix B1... 2,1To b 2,32 As the second input data, and b 2,i Send it to the i-th multiplier 421, for example, send b 2,1 Send it to the first multiplier 421, and b 2,2 Send it to the second multiplier 421, and b 2,32 The i-th multiplier 421 is sent to the 32nd multiplier 421, and the i-th multiplier 421 calculates a² × b. 2,i Then, a2×b 2,i The data is sent to the i-th accumulator 422 connected to it, and the i-th accumulator 422 calculates a1×b. 1,i +a2×b 2,i Following the above calculation method, the above calculation process is performed on each element in the first matrix A1 until the i-th accumulator 422 calculates the accumulation result corresponding to each column in the second matrix A2.

[0072] It should be understood that when the number of columns in the second matrix is ​​greater than m, the third selector 413 first selects elements from the first m columns of the second matrix as the second input data to the multiplier 421 for operation. Specifically, after processing the first m elements of the first row, the first m elements of the second row are processed and added to the result of the first row, and so on, until the first m columns of the second matrix are accumulated. After the accumulated result is output, each accumulator is cleared. Then, the third selector 413 selects elements from the (m+1)th to the 2mth columns of the second matrix as the second input data to the multiplier 421 for operation, and the specific operation process is the same as that for the first m columns. The second matrix is ​​divided into multiple sub-matrices by columns, and each sub-matrix is ​​processed in the above manner until all elements in the second matrix are processed. If the number of columns with unprocessed elements in the second matrix is ​​less than m, then all unprocessed elements in each row are selected as the second input data to the multiplier 421 for operation.

[0073] by Figure 7 Taking the second matrix B1 as an example, the first 32 elements of each row are selected sequentially as the second input data for processing. The results of the processed rows in the first 32 columns are accumulated until the first 32 elements of the 32nd row are used as the second input data for processing. The processing result of the first 32 elements of the 32nd row is then accumulated with the result of the first 31 rows in the first 32 columns. After outputting the accumulated result, the 32 accumulators 422 are cleared. Then, the last 32 elements of each row are selected as the second input data for processing, and the results of the processed rows in the last 32 columns are accumulated until the last 32 elements of the 32nd row are used as the second input data for processing. The processing result of the last 32 elements of the 32nd row is then accumulated with the result of the first 31 rows in the last 32 columns. After outputting the accumulated result, the 32 accumulators 422 are cleared.

[0074] In this embodiment of the application, in the dense operation mode, the first selector 411 selects an element from the first matrix and sends it to the second selector 412. The second selector 412 selects the element from the first selector 411 as the first input data. At the same time, the third selection data 413 selects q elements from the second matrix corresponding to the element selected by the first selector 411 and distributes them to q multipliers. Each multiplier 411 is responsible for calculating the product of each element in the corresponding column of the second matrix and each element in the first matrix, thus realizing matrix multiplication operation of non-sparse matrices.

[0075] In one possible implementation, such as Figure 6 As shown, the control subunit 410 further includes a fourth selector 414, a fifth selector 415, and m sixth selectors 416. The fourth selector 414 is connected to each of the sixth selectors 416, the fifth selector 415 is connected to each of the sixth selectors 416, each sixth selector 416 is connected to a second selector 412, and different sixth selectors 416 are connected to different second selectors 412.

[0076] The third selector 413 can select q elements from the fourth matrix as second input data in each operation cycle, and distribute the selected q second input data to q multipliers 421 sequentially. The fourth selector 414 can select q index values ​​corresponding to the q elements selected by the third selector 413 in each operation cycle, and distribute the selected q index values ​​to q sixth selectors 416, where the index values ​​indicate the position of the corresponding element in the fourth matrix before sparsification. The fifth selector 415 can select n elements from the third matrix and send them to each sixth selector 416, where n is a positive integer less than m. The sixth selector 416 can select the element whose position in the third matrix matches the received index value from the received n elements in each operation cycle and send it to the connected second selector 412. The second selector 412 can, in sparse operation mode, send the element sent by the connected sixth selector 416 as first input data to the connected multiplier 421.

[0077] The fourth selector 414 can select an index value from either the first or the second cache. When the fourth selector 414 selects an index value from the first cache, the index value matrix formed by the index values ​​reuses the first cache with the third matrix. When the fourth selector 414 selects an index value from the second cache, the index value matrix formed by the index values ​​reuses the second cache with the fourth matrix.

[0078] In dense operation mode, the second selector 412 can select an element from the first selector 411 as the first input data based on the received control signal. In sparse operation mode, the second selector 412 can select an element from the sixth selector 416 as the first input data based on the received control signal.

[0079] The fifth selector 415 selects n elements from the third matrix each time and sends them to each sixth selector 416. The value of n is determined by the number of rows included in each group during the sparsification process of the matrix. During structured sparsification of the matrix, adjacent n rows of elements in the matrix are grouped together. One or more elements in each column of each group are sparsed to zero according to a preset sparsification ratio. The zero-element elements are then removed to form a new matrix, thus achieving structured sparsity. For example, when obtaining the fourth matrix through structured sparsification, adjacent 4 rows of elements are grouped together, so n equals 4. That is, the fifth selector 415 selects 4 adjacent elements from the third matrix each time and sends them to each sixth selector 416. Based on the above explanation, the fifth selector 415 selects n elements from the third matrix each time and sends them to each sixth selector 416, ensuring that the n elements selected by the fifth selector 415 include the first input data that matches the position of the second input data received by the corresponding multiplier 421, thereby ensuring that matrix multiplication can proceed smoothly.

[0080] The third selector 413 selects q elements from the fourth matrix as second input data in each operation cycle, where q is a positive integer less than or equal to m. Specifically, if the number of columns in the fourth matrix is ​​greater than m, the third selector 413 first selects elements from the first m columns of the fourth matrix as second input data in each operation cycle and feeds them to the multiplier 421 for operation. Specifically, after processing the first m elements of the first row, the first m elements of the second row are processed and added to the result of the first row, and so on, until the first m columns of the second matrix are accumulated. After outputting the accumulated result, each accumulator is cleared. Then, the third selector 413 selects elements from the (m+1)th to the 2mth columns of the fourth matrix as second input data and feeds them to the multiplier 421 for operation, following the same processing method as the first m columns. The fourth matrix is ​​divided into multiple sub-matrices by columns, and each sub-matrix is ​​processed sequentially in the above manner until all elements in the fourth matrix have been processed. It should be understood that if the number of columns of unprocessed elements in the fourth matrix is ​​less than m, then all unprocessed elements in each row are selected in turn as the second input data and fed into the multiplier 421 for operation.

[0081] Since the fourth matrix is ​​a sparsified matrix, the positions of each element in the fourth matrix have changed compared to their positions in the matrix before sparsification. To ensure that the sixth selector 416 can select the first input data corresponding to the position of the second input data, the fourth selector 414 selects the index value corresponding to the second input data selected by the third selector 413 and sends the index value to the corresponding sixth selector 416. Since the index value indicates the position of the second input data in the matrix before sparsification, the sixth selector 416 can select the element that matches the second input data from the received n elements as the first input data based on the received index value, thus ensuring the accuracy of the matrix multiplication result.

[0082] Figure 8 This is a schematic diagram of matrix sparsity according to an embodiment of this application. Figure 8 As shown, matrix B2 is structured sparse to obtain matrix B2ˊ. The index matrix B2ˊˊ records the index values ​​of each element in matrix B2ˊ. An index value of 2ˊb00 indicates that the corresponding element is in the first row of matrix B2, an index value of 2ˊb01 indicates that the corresponding element is in the second row, an index value of 2ˊb10 indicates that the corresponding element is in the third row, and an index value of 2ˊb11 indicates that the corresponding element is in the fourth row. For example, if element B32 in matrix B2ˊ has an index value of 2ˊb10 and is located in the second column of matrix B2ˊ, then element B12 is located in the third row and second column of matrix B2.

[0083] For example, after the third selector 413 selects element B32 in matrix B2ˊ as the second input data, the fourth selector 414 selects the index value 2ˊb10 of element B32 and sends it to the corresponding sixth selector 416. The sixth selector 416 selects the element located in the third column of the third matrix from the four received elements according to the index value 2ˊb10 as the first input data.

[0084] In this embodiment, the third selector 413 selects the second input data from the sparsified fourth matrix, and the fourth selector 414 selects the index value indicating the position of the second input data in the matrix before sparsification and sends it to each of the sixth selectors 416. The sixth selector 416 selects the element in the third matrix that matches the position of the second input data as the first input data according to the index value. Then, the multiplier 421 calculates the product of the first input data and the second input data that match the position. Under the premise of ensuring the accuracy of the matrix multiplication operation result, the matrix multiplication operation of the sparsified matrix is ​​realized, so that the matrix multiplication calculation unit 611 can be used for matrix multiplication operations of non-sparse matrices and matrix multiplication operations of sparse matrices, ensuring that the matrix multiplication calculation unit 611 has strong applicability.

[0085] In one possible implementation, when the sparsity ratio of the fourth matrix is ​​p:n, the fifth selector 415 can sequentially select n elements from the third matrix every p operation cycles and send them to each sixth selector 416, where p is a positive integer less than n.

[0086] When obtaining the fourth matrix through structured sparsity, different sparsity ratios can be used. For example, when performing structured sparsity by grouping adjacent 4 rows into a single group, the sparsity ratio can be 1:4 or 2:4. Using different sparsity ratios for structured sparsity will result in different numbers of rows in the fourth matrix. When the sparsity ratio is 1:4, the fourth matrix has 1 row, and when the sparsity ratio is 2:4, the fourth matrix has 2 rows. Figure 8 This illustrates the matrix sparsification process when the sparsification ratio is 1:4. Figure 9 The process of matrix sparsification is shown when the sparsification ratio is 2:4.

[0087] like Figure 9 As shown, by performing structured sparsity on matrix B3, matrix B3ˊ is obtained. The index matrix B3ˊˊ records the index values ​​of each element in matrix B3ˊ. The index value 2ˊb00 indicates that the corresponding element is located in the first row of matrix B3, the index value 2ˊb01 indicates that the corresponding element is located in the second row of matrix B3, the index value 2ˊb10 indicates that the corresponding element is located in the third row of matrix B3, and the index value 2ˊb11 indicates that the corresponding element is located in the fourth row of matrix B3.

[0088] When p equals 1 (i.e., the sparsity ratio of the fourth matrix is ​​p:n), the fourth matrix has 1 row. In each operation cycle, the third selector 413 selects q elements from the fourth matrix as the second input data. The fifth selector 415 can select elements whose positions match those selected by the third selector 413 as the first input data in one operation cycle. Therefore, the fifth selector 415 can select n elements from the third matrix in each operation cycle and send them to each sixth selector 416. When p is greater than 1, the fourth matrix has more than 1 row. In each operation cycle, the third selector 413 selects q elements in the same row from the fourth matrix as the second input data. However, some of the first input data selected by the fifth selector 415 in one operation cycle do not match the positions of the second input data selected by the third selector 413, requiring the third selector 413 to select the second input data again in the next operation cycle. Therefore, corresponding to the third selector 413 selecting q elements from the fourth matrix in each operation cycle, the fifth selector 415 selects n elements from the third matrix every p operation cycles and sends them to each sixth selector 416.

[0089] In this embodiment, the fifth selector can select the first input data from the third matrix according to different data retrieval periods based on the sparsity ratio of the fourth matrix, thereby enabling matrix multiplication operations on matrices after structured sparsification with different sparsity ratios, further improving the applicability of the matrix multiplication calculation unit 611.

[0090] In one possible implementation, such as Figure 3 and Figure 4 As shown, the control subunit 410 can read first input data from the first buffer 233 and send it to the arithmetic subunit 420, and read second input data from the second buffer 234 and send it to the arithmetic subunit 420. The arithmetic subunit 420 can store the calculation result in the third buffer 235. In addition, the control subunit 410 can also read the index value from the first buffer 233.

[0091] In this embodiment, the first input data and the second input data are read from different buffers, which ensures the efficiency of data reading and the logical correctness of data reading, thereby ensuring the efficiency and accuracy of matrix multiplication operations.

[0092] Matrix multiplication computation method

[0093] Figure 10 This is a flowchart of a matrix multiplication calculation method according to an embodiment of this application. This matrix multiplication calculation method can be applied to the matrix multiplication calculation unit 611 in any of the above embodiments. For example... Figure 10 As shown, the matrix multiplication calculation method includes the following steps:

[0094] Step 1001: Receive control signals;

[0095] Step 1002: Determine the dense operation mode or the sparse operation mode according to the control signal. In the dense operation mode, select the data in the first matrix as the first input data and select the data in the non-sparsed second matrix as the second input data. In the sparse operation mode, select the data in the third matrix as the first input data and select the data in the sparsed fourth matrix as the second input data.

[0096] Step 1003: Perform matrix multiplication based on the first input data and the second input data, and output the result.

[0097] Since the details of the matrix multiplication calculation method described above have been explained in detail in conjunction with the structural diagram in the matrix multiplication calculation unit section of the above embodiments of this application, the specific process can be found in the description of the aforementioned matrix multiplication calculation unit embodiments, and will not be repeated here.

[0098] Commercial value of embodiments of the present application

[0099] In addressing the technical problem of poor applicability of matrix multiplication calculation units, this application employs a control subunit composed of multiple selectors. This control subunit can operate in either dense or sparse operation mode based on control signals. Different data retrieval logics are used in either dense or sparse operation mode. In dense operation mode, matrix multiplication of non-sparse matrices is performed, while in sparse operation mode, matrix multiplication of sparse matrices is performed. This allows for the simultaneous fulfillment of the operational logic of both structured sparse neural networks and dense neural networks, resulting in a matrix multiplication calculation unit with strong applicability.

[0100] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the method embodiments, since they are basically similar to the methods described in the apparatus and system embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions of other embodiments.

[0101] It should be understood that the foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] It should be understood that the use of a singular form to describe an element or to show only one element in the accompanying drawings does not imply that the number of such element is limited to one. Furthermore, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be broken down into multiple modules or elements.

[0103] It should also be understood that the terminology and expressions used herein are for descriptive purposes only, and one or more embodiments described herein should not be limited to these terms and expressions. The use of these terms and expressions does not exclude any illustrative and descriptive equivalent features (or parts thereof), and it should be recognized that various modifications that may exist should also be included within the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be considered to cover all such equivalents.

Claims

1. A matrix multiplication computing device, comprising: The control subunit and the operation subunit; The control subunit is configured to operate in a dense operation mode or a sparse operation mode according to the received control signal, wherein in the dense operation mode, data in a first matrix is selected as first input data and sent to the operation subunit, and data in a second matrix that is not sparsified is selected as second input data and sent to the operation subunit, and in the sparse operation mode, data in a third matrix is selected as first input data and sent to the operation subunit, and data in a fourth matrix that is sparsified is selected as second input data and sent to the operation subunit; The operation subunit is configured to perform matrix multiplication operation according to the received first input data and second input data, and output operation result; The operation subunit comprises m multipliers and m accumulators, m being a positive integer greater than or equal to 2, each of the multipliers is connected with one of the accumulators, and different multipliers are connected with different accumulators; The multiplier is configured to calculate the product of the received first input data and second input data, and send the product to the connected accumulator; The accumulator is configured to accumulate and sum the received product; The control subunit comprises a first selector, m second selectors and a third selector, the first selector is connected with each of the second selectors, each of the second selectors is connected with one of the multipliers, and different second selectors are connected with different multipliers, and the third selector is connected with each of the multipliers; The first selector is configured to sequentially select one element from the first matrix as first input data and send the element to each of the second selectors in each operation period; The second selector is configured to send the element sent by the first selector as first input data to the connected multiplier in the dense operation mode; The third selector is configured to select q elements corresponding to the element selected by the first selector from the second matrix as second input data in each operation period, and sequentially distribute the q second input data to q multipliers, q being a positive integer less than or equal to m.

2. The matrix multiplication computing device of claim 1, wherein, The control subunit further comprises a fourth selector, a fifth selector and m sixth selectors, the fourth selector is connected with each of the sixth selectors, the fifth selector is connected with each of the sixth selectors, each of the sixth selectors is connected with one of the second selectors, and different sixth selectors are connected with different second selectors; The third selector is configured to select q elements from the fourth matrix as second input data in each operation period, and sequentially distribute the q second input data to q multipliers; the fourth selector is configured to select, in each operation cycle, q index values corresponding to the q elements selected by the third selector, and distribute the q index values to q sixth selectors, the index values being used to indicate positions of corresponding elements in the fourth matrix in a matrix before being sparsified; the fifth selector is configured to select n elements from the third matrix and send the n elements to each of the sixth selectors respectively, n being a positive integer less than m; the sixth selector is configured to select, in each operation cycle, an element from the n elements received and having a position in the third matrix matching the index value received, and send the element to the second selector connected to the sixth selector; the second selector is configured to send, in the sparse operation mode, the element sent by the sixth selector connected to the second selector as first input data to the multiplier connected to the second selector.

3. The matrix multiplication calculation device according to claim 2, wherein the fifth selector is configured to select n elements from the third matrix and send the n elements to each of the sixth selectors respectively in each p operation cycles, p being a positive integer less than n, when the sparsification ratio of the fourth matrix is p:n.

4. The matrix multiplication calculation device according to any one of claims 1-3, wherein the control subunit is configured to read first input data from the first buffer and send the first input data to the operation subunit, and read second input data from the second buffer and send the second input data to the operation subunit; the operation subunit is configured to store the operation result in the third buffer.

5. An acceleration device, comprising: the matrix multiplication calculation device according to any one of claims 1-4; a main control unit configured to control the matrix multiplication calculation device to work.

6. The acceleration device of claim 5, further comprising: a data buffer unit, wherein the data buffer unit comprises at least a first buffer, a second buffer and a third buffer; the first buffer is configured to buffer the first matrix or the third matrix; the second buffer is configured to buffer the second matrix or the fourth matrix; the third buffer is configured to buffer an operation result output by the operation subunit.

7. The acceleration device of claim 6, wherein, The first buffer, the second buffer and the third buffer are all first-in-first-out buffers.

8. A neural network accelerator, comprising: the acceleration device according to any one of claims 5-7; a scheduling unit configured to schedule the acceleration device to obtain a first matrix and a second matrix that is not sparsified, and schedule the acceleration device to perform matrix multiplication operation on the first matrix and the second matrix, or schedule the acceleration device to obtain a third matrix and a fourth matrix that is sparsified, and schedule the acceleration device to perform matrix multiplication operation on the third matrix and the fourth matrix.

9. A computing system comprising: a processor, a direct memory access controller, a communication interface, a static random access memory, a communication bus and the neural network accelerator according to claim 8; the processor, the direct memory access controller, the communication interface and the static random access memory complete communication with each other through the communication bus; the processor, the direct memory access controller, the communication interface and the static random access memory complete communication with each other through the communication bus. The processor, the direct memory access controller, the communication interface and the neural network accelerator complete communication with each other through a communication bus; The static random access memory and the neural network accelerator exchange data through a private interface; The processor is used for control of the computing system and initialization of the neural network accelerator.

10. A matrix multiplication calculation method applied to the matrix multiplication calculation device according to any one of claims 1-4, comprising: receiving a control signal; determining a dense operation mode or a sparse operation mode according to the control signal, in the dense operation mode, data in a first matrix is selected as first input data, and data in a non-sparse second matrix is selected as second input data, in the sparse operation mode, data in a third matrix is selected as first input data, and data in a sparse fourth matrix is selected as second input data; performing matrix multiplication operation according to the first input data and the second input data, and outputting operation result.

Images