Matrix Multiplication Hardware Architecture

Abstract

To provide a matrix multiplication hardware architecture.SOLUTION: A matrix multiplication hardware architecture includes: a reduction network including a tree topology with multiple levels formed by a plurality of reduction network nodes, each of which includes a data selector and two computation paths; and a chain formed by cascading a plurality of digital signal processing units (DSPs), where output ends of adjacent DSPs are respectively connected to the two computation paths of the same reduction network node in a first level of the tree topology, and outputs of the two computation paths are connected to a reduction network node in a higher level of the tree topology through data selectors. An addition tree of the FSB is changed into an addition chain adapted to the DSP structure, so that rear adders of the DSPs are multiplexed, the hardware utilization rate is improved, and the sign extension of higher-order bits can also be changed into zero padding.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to the technical field of digital signal processing, and specifically to matrix multiplication hardware architecture. [Background technology]

[0002] In large language models based on the Transformer algorithm, matrix-matrix multiplication (General Matrix Multiplication, GEMM) is widely applied in technical fields such as solving complex physical systems, calculating current distribution in circuits and analyzing engineering problems, processing multidimensional data, analyzing social networks, movie recommendation systems, and transportation planning and management. This is the most important and time-consuming computational operation. To reduce the computational load and improve computational efficiency, optimization methods such as sparsification must be adopted and efficient hardware architectures must be designed to speed up matrix multiplication operations. Sparsification has become a widely used method for accelerating GEMM, and related dedicated hardware architectures have been implemented.

[0003] For matrices with block-based flexible sparsity, the existing flexible sparse block hardware computing architecture (Flexible Sparse Block, FSB) consists of multiple multipliers and a dynamically scalable reduction network, where the dynamically scalable reduction network is composed of an adder tree and configuration logic. As shown in Figure 1, Figure 1(a) shows the process of reading an array from storage in the flexible sparse block hardware computing architecture, which includes steps such as defining a memory address, loading array data, reading array data, and processing the array data. The step of defining a memory address uses a register to store binary code. Figure 1(b) shows the decode input process in the flexible sparse block hardware computing architecture, which means decoding the input signal or data to extract useful information or data from it. This includes receiving input data, analyzing the data format, decoding the data, and then decoding the decoded data. The flexible sparse block hardware computing architecture uses a selector to filter or transform the data. Figure 1(c) shows the reduction network configuration process of the flexible sparse block hardware computing architecture, which realizes data transmission and optimization based on a pre-configured network topology. Figure 1(d) shows the data forwarding process of the flexible sparse block hardware computing architecture, which also realizes hierarchical multiplexed time-sharing data transmission based on a pre-configured network topology, and factors such as hardware resource usage and energy consumption must be taken into consideration. Figure 1(e) shows the calculation and reduction process of the flexible sparse block hardware computing architecture, which uses a reduction method based on vector-matrix multiplication and a tree structure. Figure 1(f) shows the accumulation and addition process of the flexible sparse block hardware computing architecture.

[0004] Depending on the sparseness of the current block, the reduce nodes in each layer are configured to perform an addition or forward propagation function, i.e., the selectors choose to output either the adder result or the bit-splice result.

[0005] However, the hardware computing architecture in related technologies such as FSB has the following drawbacks:

[0006] Hardware optimization for field-programmable gate arrays (FPGAs) is insufficient, which is manifested in the low computational efficiency of the addition tree structure and the high resource consumption of FPGAs. If the hardware parallelism is p, the computation unit requires p multipliers and (p / 2^1 + p / 2^2 + p / 2^3 +... + 1) adders. To speed up the inference process of large language models, multiple computation units must be repeatedly configured to achieve high computational efficiency, which consumes a large amount of the FPGA's limited look-up table (LUT) resources. Summary of the Invention

[0007] In response to the problems in the related art, the present invention provides a matrix multiplication hardware architecture that can significantly save resources and optimize timing.

[0008] An embodiment of the present invention provides a matrix multiplication hardware architecture, a reduction network including a multi-level tree topology formed by a plurality of reduction network nodes, each of which includes a data selector and two computation paths; The present invention also includes a digital signal processing unit DSP48 chain cascaded by a plurality of digital signal processing units DSP48, wherein the output terminals of adjacent digital signal processing units DSP48 are respectively connected to two calculation paths of the same reduction network node in the first level tree topology, and the outputs of the two calculation paths are connected to the reduction network node in the upper level tree topology via a data selector.

[0009] Furthermore, the digital signal processing unit DSP48 includes four input ports for receiving sparse matrix data and dense matrix data, and output ports connected to adjacent cascaded digital signal processing units DSP48.

[0010] Furthermore, each of the four input ports has: Input port B, input port A and input port D for receiving sparse and dense matrix data; an input port C for connection to the output of a higher level digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain; and An output port P for connection to the input of the next level digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain.

[0011] Furthermore, the digital signal processing unit DSP48 includes a pre-adder, a post-adder, multiple sets of logic circuits provided on the input and output sides of the pre-adder, and multiple sets of logic circuits provided on the input and output sides of the post-adder.

[0012] Furthermore, the plurality of sets of logic circuits include: a logic circuit connected to two output ports of the digital signal processing unit DSP48 for connecting to the pre-adder; a logic circuit connected to another output port of the digital signal processing unit DSP48, used to connect to the post-adder and simultaneously connect the post-adder to the output of the pre-adder; and a logic circuit used at the output end of the digital signal processing unit DSP48.

[0013] Furthermore, the digital signal processing unit DSP 48 is configured to be able to calculate multiple 8-bit multiplications simultaneously.

[0014] Furthermore, the parallelism of the digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain is an integer multiple of four, and each parallelism corresponds to matrix multiplication calculations of various sparse formats.

[0015] Furthermore, the two computation paths are used for the addition operation and the splice function on the input data of the reduction network node, respectively.

[0016] Furthermore, the data selector is used to select and output one of two calculation paths based on a preset selection signal.

[0017] Furthermore, the preset selection signal of the data selector is determined based on the sparseness of the input matrix, the parallelism of the digital signal processing unit DSP 48, and the depth in the multi-level tree topology where the reduction network node is located.

[0018] Other optional features and technical advantages of embodiments of the present invention will be described in part below and in part will be apparent from reading this specification.

[0019] Compared with the prior art, the present invention has the following beneficial technical effects:

[0020] The present invention provides a matrix multiplication hardware architecture, including a reduction network with a multi-level tree topology formed by multiple reduction network nodes, each of which includes a data selector and two calculation paths; and a DSP48 chain cascaded by multiple DSP48 digital signal processing units, where the output terminals of adjacent DSP48 digital signal processing units connect to two calculation paths of the same reduction network node in the first-level tree topology, and the outputs of the two calculation paths connect to a reduction network node in a higher-level tree topology via a data selector. The present invention modifies the addition tree of the FSB into an addition chain compatible with the DSP48 structure, thereby multiplexing the post-adders of the DSP48 and improving hardware utilization. At the same time, the present hardware architecture modifies the sign extension of the upper bits to zero-filling, thereby significantly saving resources and optimizing timing. [Brief explanation of the drawings]

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the elements shown in the drawings are not limited to the scale shown in the drawings, and the same or similar reference numerals in the drawings indicate the same or similar elements. [Figure 1] 1 shows a schematic diagram of a flexible sparse block hardware computing architecture in the related art. [Figure 2] 1 shows a schematic diagram of a matrix multiplication hardware architecture in accordance with an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in more detail below in conjunction with the embodiments and drawings, wherein the exemplary embodiments of the present invention and the description thereof are for illustrating the present invention but not for limiting the present invention.

[0023] As used herein, the term "comprises" and variations thereof refer to an open inclusion, i.e., "including, but not limited to." The term "or" means "and / or" unless otherwise noted. The term "based on" means "based at least in part on." The terms "one exemplary embodiment" and "one example" mean "at least one exemplary embodiment." The term "another example" means "at least one other example." Terms such as "first," "second," etc. may refer to different or identical things. The following may include other explicit and implicit definitions.

[0024] FIG. 2 shows a matrix multiplication hardware architecture in an embodiment of the present invention. As shown in FIG. 2, the matrix multiplication hardware architecture according to the present invention includes: a reduction network including a multi-level tree topology formed by a plurality of reduction network nodes, each of which includes a data selector and two computation paths; The present invention also includes a digital signal processing unit DSP48 chain cascaded by a plurality of digital signal processing units DSP48, wherein the output terminals of adjacent digital signal processing units DSP48 are respectively connected to two calculation paths of the same reduction network node in the first level tree topology, and the outputs of the two calculation paths are connected to the reduction network node in the upper level tree topology via a data selector.

[0025] It should be noted that the multi-level tree topology described in this embodiment is a local area network topology similar to a bus topology, which is composed of a tree structure and has the characteristics of a tree structure. In the tree topology, the tree network may include branches, and each branch may include multiple reduction network nodes. The tree topology is an extended form of the bus topology, and the transmission medium is a branch circuit in which the transmission medium is not closed.

[0026] The tree topology has a root node and branch nodes, making it suitable for hierarchical structures and hierarchical management systems that prioritize and grade data. Similar to bus topology, tree topology allows one site to transmit data and other sites to receive it. Furthermore, tree topology is highly collapsible, effectively protecting cabling investments.

[0027] In the multi-level tree topology structure and the digital signal processing unit DSP48 cascade in this embodiment, as shown in FIG. 2, there are a plurality of multiplexers (MUX) for data signal processing and transmission between the digital signal processing unit DSP48 of the upper and lower levels and the reduction network node. Specifically, the multiplexer (MUX) is a multiplexer for integrating multiple input lines of multiple signals into one output line. The multiplexer (MUX) has a set of specific input terminals, each of which may have one or more input signals or no signals, and a set of specific input terminals for input signals in advance. It includes one selection terminal that needs to be selected and one set of output terminals. The multiplexer (MUX) only has one output port and all signals need to be combined into this one output port. The multiplexer (MUX) only outputs the selected input signal and ignores the others. The main working principle of the multiplexer (MUX) is that when there is a signal input at the input terminal, it will combine the corresponding input signal into the output terminal based on the input selection signal, and all signals at the other input terminals will be ignored. Therefore, the multiplexer (MUX) can effectively combine multiple signals, thereby saving output line resources and reducing system costs.

[0028] In this embodiment, the digital signal processing unit DSP48 includes four input ports for receiving sparse matrix data and dense matrix data, and an output port connected to adjacent cascaded digital signal processing units DSP48, specifically, the four input ports respectively: Input port B, input port A and input port D for receiving sparse and dense matrix data; an input port C for connection to the output of a higher level digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain; and An output port P for connection to the input of the next level digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain.

[0029] More specifically, in this embodiment, a DSP48 is employed, whose input port B is 18 bits wide, whose input port A is 30 bits wide, and whose input port D is the data port of the pre-adder and is 25 bits wide.

[0030] In this embodiment, the digital signal processing unit DSP48 is provided with a pre-adder, a post-adder, a plurality of sets of logic circuits provided on the input and output sides of the pre-adder, and a plurality of sets of logic circuits provided on the input and output sides of the post-adder. Specifically, the plurality of sets of logic circuits are a logic circuit connected to two output ports of the digital signal processing unit DSP48 and connected to a pre-adder; a logic circuit connected to another output port of the digital signal processing unit DSP48, used to connect to a post-adder, and simultaneously connecting the post-adder to the output terminal of the pre-adder; and a logic circuit used at the output end of the digital signal processing unit DSP48.

[0031] Specifically, the logic circuit generally includes parts such as an input interface, an arithmetic unit, control logic, and an output interface. The input interface receives data signals input from the outside and converts them into a format suitable for internal calculation. The arithmetic unit is the core of the digital signal processing unit DSP48, and includes multipliers, adders, shifters, etc., and is used to execute various digital signal processing algorithms. The control logic controls the workflow of the arithmetic unit and ensures the correct calculation order and result output. The output interface outputs the processed data signals to an external device or memory. In addition, the logic circuit of the digital signal processing unit DSP48 needs to consider the interface and communication protocol with the external device to ensure correct communication and data transmission with the external device.

[0032] In this embodiment, the digital signal processing unit DSP48 is configured to perform multiple 8-bit multiplications simultaneously. The digital signal processing unit DSP48 generally includes multiple multipliers that can simultaneously perform multiple 8-bit multiplications. When performing a multiplication, the digital signal processing unit DSP48 multiplies the input data by the coefficients of the multipliers and accumulates the results. The multipliers operate in parallel, allowing multiple multiplications to be performed simultaneously, thereby achieving multiple 8-bit multiplications simultaneously.

[0033] In this embodiment, the parallelism of the digital signal processing units DSP48 in the digital signal processing unit DSP48 chain is an integer multiple of 4, and each parallelism supports matrix multiplication calculations of various sparse formats. Specifically, the digital signal processing units DSP48 in the digital signal processing unit DSP48 chain may be composed of 4, 8, or 16 parallel DSPs, that is, the parallelism of the digital signal processing unit DSP48 is 4, 8, or 16, and the calculation units of each DSP parallelism may support matrix multiplication calculations of multiple sparse formats. For example, a calculation unit with a DSP parallelism of 16 can support sparse matrix calculations of 1:16, 2:16, 4:16, or 8:16 in total, and can also complete dense matrix multiplication at the same time.

[0034] In this embodiment, the two calculation paths are respectively used for the addition operation and the joining function on the input data of the reduction network node, and specifically, the addition operation of the calculation path has the roles of the accumulation function and the filter function.

[0035] Regarding the accumulation function, in digital signal processing, accumulation is a common operation to calculate the sum or average value of a signal. The digital signal processing unit DSP 48 can perform an accumulation operation on the input data through an addition operation to obtain the desired result.

[0036] Regarding the filtering function, addition operation plays an important role in the digital filter. The digital signal processing unit DSP 48 can realize signal filtering by adding the input data and the filter coefficients, thereby removing noise or extracting specific frequency components.

[0037] The splice function of the computational path is responsible for data merging and resolution enhancement.

[0038] Regarding data merging, the splice function allows multiple input data to be combined into one larger data block. This is very useful when processing segmented signals or when multiple signals need to be combined into one signal. The splice operation allows the digital signal processing unit DSP48 to process longer data sequences, thereby improving processing efficiency.

[0039] Regarding resolution improvement, by splicing multiple input data, the resolution of the data can be increased. This is especially important in applications such as image processing and audio processing. By splicing multiple 8-bit data, data with a higher number of bits can be obtained, thereby improving the accuracy and quality of the processing.

[0040] Specifically, in this embodiment, the data width is expanded after passing through the reduction network node, and before the digital signal processing unit DSP48 performs the matrix multiplication calculation, the selection signal configures the data stream inside the digital signal processing unit DSP48 according to the sparsity of the matrix, so that the present application can flexibly respond to various sparsity calculations.

[0041] In this embodiment, the data selector is used to select and output one of two operation paths based on a preset selection signal; The data selector is used in the digital signal processing unit DSP 48 to selectively receive and process multiplexed data as needed, realizing multiplexed time-division transfer of data and logic control, thereby improving the processing efficiency and functional flexibility of the digital signal processing unit DSP 48. Specifically, it is used for data selection function, data time-division transfer, and logic control.

[0042] Regarding the data selection function, the data selector can select a specified one from a set of input signals based on a given input address code and send it to the output end, thereby allowing the digital signal processing unit DSP48 to optionally receive and process multiplexed data as required.

[0043] Regarding data time division transmission, during the process of multiplexed data transmission, the data selector can select any one of them as needed, so that the digital signal processing unit DSP48 can realize multiplexed time division transmission of data and improve data processing efficiency.

[0044] Regarding logic control, the data selector, as part of logic control, can realize certain logic functions by selecting different input signals, which is very useful in digital signal processing and can help the digital signal processing unit DSP48 realize various complex logic operations and controls.

[0045] In this embodiment, the preset selection signal of the data selector is determined based on the sparseness of the input matrix, the parallelism of the digital signal processing unit DSP48, and the depth of the reduction network node in the multi-level tree topology, where the depth in the multi-level tree topology means the number of stages in which the reduction network node is located in the multi-level tree topology.

[0046] As shown in FIG. 2, taking an arithmetic unit consisting of four digital signal processing units DSP48 as an example, the four digital signal processing units DSP48 include digital signal processing unit DSP480, digital signal processing unit DSP481, digital signal processing unit DSP482, and digital signal processing unit DSP483. The arithmetic operation for a set of 4*4 matrix input data with a sparseness of 2:4 is as follows:

[0047] Four 8-bit data A, B, C, and D from the sparse matrix are sent to input port B of digital signal processing unit DSP48-0, digital signal processing unit DSP48-1, digital signal processing unit DSP48-2, and digital signal processing unit DSP48-3, respectively; four 8-bit data a, b, c, and d from the dense matrix are sent to input port A of digital signal processing unit DSP48-0, digital signal processing unit DSP48-1, digital signal processing unit DSP48-2, and digital signal processing unit DSP48-3, respectively; and similarly, four 8-bit data a', b', c', and d' from the dense matrix are sent to input port D of digital signal processing unit DSP48-0, digital signal processing unit DSP48-1, digital signal processing unit DSP48-2, and digital signal processing unit DSP48-3, respectively.

[0048] One digital signal processing unit, DSP48, We can calculate (A+D)*B+C=(A*B+C1)+(D*B+C2), that is, two multiplication operations, where the data on input port A and input port D come from different big model input sequence length (seq_len) dimensions.

[0049] The output result from the output port P of the upper level digital signal processing unit DSP48 is transmitted to the input port C of the lower level digital signal processing unit DSP48; The output result of the output port P of the digital signal processing unit DSP48 consists of two parts, including the output result and the zero splice configuration, (A*B+C1) and (D*B+C2), respectively. For each of the two digital signal processing units DSP48, these two output signals are sent to the first-level splicer, which zero-pads the input signal to 32 bits, and the output result of the first-level splicer is sent to the second-level splicer, which is spliced ​​into a 64-bit output result.

[0050] The computational architecture of the sparseness-configurable digital signal processing unit DSP48 in this embodiment includes two parts: a runtime reconfigurable DSP cascade chain and a configurable reduction network. The FSB's addition tree can be changed into an addition chain that is compatible with the DSP48's structure, thereby multiplexing the post-adders of the DSP48 and improving hardware utilization. At the same time, the sign bit extension of the upper bits can be changed to fill 0, thereby significantly saving resources and optimizing timing.

[0051] Although multiple embodiments of the present invention have been described herein, for the sake of clarity, the description of each embodiment is not complete, and identical or similar features or portions may be omitted among the embodiments. In this specification, the terms "one embodiment," "several embodiments," "example," "specific example," or "several examples" mean that they apply to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily refer to the same embodiment or example. Those skilled in the art can combine and combine different embodiments or examples, as well as features of different embodiments or examples, described herein, if they are not mutually inconsistent.

[0052] While the exemplary systems and methods of the present invention have been specifically shown and described with reference to the above-described embodiments, these are merely examples of the best modes for carrying out the systems and methods. Those skilled in the art will recognize that various modifications can be made to the embodiments of the systems and / or methods in practicing the systems and / or methods without departing from the spirit and scope of the present invention as defined in the claims.

Claims

1. 1. A matrix multiplication hardware architecture, comprising: a reduction network including a multi-level tree topology formed by a plurality of reduction network nodes, each of which includes a data selector and two computation paths; a digital signal processing unit DSP48 chain cascaded by a plurality of digital signal processing units DSP48, wherein the output terminals of two adjacent digital signal processing units DSP48 are respectively connected to two calculation paths of the same reduction network node in the first level tree topology, and the outputs of the two calculation paths are connected to the reduction network node in the upper level tree topology via a data selector; Including, The digital signal processing unit DSP48 includes a pre-adder, a post-adder, a plurality of sets of logic circuits provided on the input side and the output side of the pre-adder, and a plurality of sets of logic circuits provided on the input side and the output side of the post-adder.

1. A matrix multiplication hardware architecture comprising:

2. The digital signal processing unit DSP 48 four input ports for receiving sparse and dense matrix data; an output port connected to an adjacent cascaded digital signal processing unit DSP 48; 2. The matrix multiplication hardware architecture of claim 1, comprising:

3. Each of the four input ports is input port B, input port A and input port D for receiving sparse matrix data and dense matrix data; an input port C for connection to the output of a higher level digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain; and An output port P for connection to the input end of the next level digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain.

3. The matrix multiplication hardware architecture of claim 2.

4. The plurality of sets of logic circuits include: a logic circuit connected to two output ports of the digital signal processing unit DSP 48 for connecting to the pre-adder; a logic circuit connected to another output port of the digital signal processing unit DSP 48, used to connect the post-adder and simultaneously connect the post-adder to the output of the pre-adder; a logic circuit used at the output end of the digital signal processing unit DSP48; 2. The matrix multiplication hardware architecture of claim 1, comprising:

5. The digital signal processing unit DSP 48 is configured to be able to simultaneously calculate multiple 8-bit multiplications. Matrix multiplication hardware architecture according to any one of claims 1 to 4, characterized in that:

6. The parallelism of the digital signal processing unit DSP48 in the digital signal processing unit DSP48 chain is an integer multiple of 4, and each parallelism corresponds to matrix multiplication calculations of various sparse formats. Matrix multiplication hardware architecture according to any one of claims 1 to 4, characterized in that:

7. The two computation paths are used for the addition operation and the splice function on the input data of the reduction network node, respectively. Matrix multiplication hardware architecture according to any one of claims 1 to 4, characterized in that:

8. The data selector is used to select and output one of two calculation paths based on a preset selection signal. Matrix multiplication hardware architecture according to any one of claims 1 to 4, characterized in that:

9. The preset selection signal of the data selector is determined based on the sparseness of the input matrix, the parallelism of the digital signal processing unit DSP 48, and the depth in the multi-level tree topology in which the reduction network node is located. Matrix multiplication hardware architecture according to any one of claims 1 to 4, characterized in that:

Images