Method and system for performing matrix multiplication

Abstract

A computer system includes multiple SIMD engines and corresponding multiple output register groups. An operand A register file stores one or more operand A values, each value including multiple operand numbers. An operand B register file stores one or more operand B values, each value including multiple operand numbers. An operand A allocation circuit receives operand A values ​​from the operand A register file and selectively routes one or more of the operand numbers of the received operand A values ​​to create multiple input operand A values, which are selectively routed to the SIMD engine. An operand B allocation circuit receives one or more operand B values ​​from the operand B register file and selectively routes one or more of the operand numbers of the operand B values ​​to create multiple input operand B values, which are selectively routed to the SIMD engine.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. patent application 16 / 397,401, filed on April 29, 2019, entitled “Efficient Architectures for Deep Learning Algorithms,” which is incorporated herein by reference. Technical Field

[0003] Provides multiple energy-efficient architectures that use single instruction multiple data (SIMD) engines to perform calculations such as matrix multiplication and convolution. These operations are at the heart of various machine learning algorithms currently implemented in various applications such as computer vision, machine translation, and automatic speech recognition. Background Art

[0004] The following describes in detail the general dot product and matrix multiplication.

[0005] The dot product of two vectors can be defined as follows. Let p = [p1 p2 ... pn] and q = [q1 q2 ... qn] be two vectors of length n. The dot product of p and q, denoted by p·q, is defined as:

[0006]

[0007] Computing the dot product of two vectors of length n requires n multiplications and (n-1) additions.

[0008] The multiplication of two vectors can be defined as follows. Let A be an (mxn) matrix (i.e., A is a matrix with m rows (horizontally) and n columns (vertically)). Let a i,j Denote the element in the i-th row and j-th column of matrix A. Let B be a (nxk) matrix. Matrices A and B are represented as follows.

[0009]

[0010] Matrices A and B can be multiplied only if the dimensions of matrices A and B are compatible (i.e., if the number of columns in A is equal to the number of rows in B). The product C of matrices A and B is defined as follows.

[0011]

[0012] From equation (3), we can see that the matrix C has m rows and k columns. It should also be noted that A x B may exist, but B x A does not.

[0013] One way to think about matrix multiplication is as follows. The element in the i-th row and j-th column of matrix C is the dot product of the i-th row in matrix A and the j-th column in matrix B. If the i-th row in matrix A is represented by a i,: Indicated by, and the j-th column in matrix B is represented by b :,j Indicates that

[0014] c i，j =a i，： b ：，j (4)

[0015] Thus, the multiplication of two matrices of dimensions (mxn) and (nxk) consists of computing the (mxk) dot product of n-length vectors.

[0016] As mentioned above, computing the dot product of two vectors of length n requires n multiplications and (n-1) additions. The dot product calculation is usually implemented as a series of multiply-accumulate operations. The multiply-accumulate operation calculates the product of two numbers and adds the product to an accumulator. This can be expressed as:

[0017] c←c+a*b(5)where a and b are two numbers to be multiplied, and c is the accumulator. The hardware unit that performs multiply-accumulate operations is called a multiplier-accumulator (MAC) unit. In this document, we also refer to the multiply-accumulate operation as a MAC. Figure 1 is a block diagram of a simple MAC unit 100, which includes input operand registers 101 and 102 storing operands a and b, respectively, a multiplication circuit 103, an addition circuit 104, and an accumulator register 105 storing an accumulator value c.

[0018] As mentioned above, matrix multiplication involves computing multiple dot products. Therefore, matrix multiplication can be parallelized by employing multiple MACs. Figure 2 2 is a block diagram of a system 200 including multiple parallel MACs 201-204 for performing multiple dot products in parallel. MACs 201, 202, 203, and 204 include operand registers 211-212, 213-214, 215-216, and 217-218, respectively, multiplier circuits 221, 222, 223, and 224, respectively, adder circuits 231, 232, 233, and 234, respectively, and accumulators 241, 242, 243, and 244, respectively.

[0019] System 200 includes four parallel MACs, where MAC 201 calculates a 1,: With b :,1 The dot product of , MAC 202 calculates a 2,: With b :,1 The dot product of , MAC 203 calculates a 3,: With b :,1The dot product of , and MAC 204 calculates a 4,: With b :,1 It should be noted that although the four MACs 201-204 use different rows of matrix A, they all use the same column of matrix B. Supplying input data to the MACs 201-204 is a challenge that needs to be solved. Therefore, it will be necessary to have an efficient way to provide the required data to computational units such as MACs 201-204.

[0020] Notice, Figure 2 All MAC units 201-204 of the processor are performing the same operation, but with different inputs. Therefore, instead of providing individual instructions to each of the MACs 201-204, all MACs 201-204 can be grouped together to form a single instruction multiple data (SIMD) engine that operates in response to common instructions.

[0021] Figure 3 is a block diagram of a SIMD engine 300 that groups the operand registers 211, 213, 215, and 217 of the MACs 201-204 to form a first operand register 301, and groups the operand registers 212, 214, 216, and 218 of the MACs 201-204 to form a second operand register 302. The multiplier circuits 221-224 of the MACs 201-204 are combined to form a multiplier 321, and the addition circuits 231-234 of the MACs 201-204 are combined to form an addition circuit 331. The accumulators 241-244 of the MACs 201-204 are combined to form an accumulator 341. In this manner, the various elements of the parallel MACs 201-204 are combined to form the SIMD engine 300. It is important to note that the scalar inputs for the different MACs 201-204 are combined to form vector inputs in the SIMD engine 300. In addition, the output of the accumulator 341 of the SIMD engine 300 will also be a vector.

[0022] Note that a SIMD engine can do many more operations than a simple multiply-accumulate operation with vector inputs. Typically, a SIMD engine is designed as a block with an instruction set that allows the SIMD engine to perform different operations. Some of the operations that a SIMD engine can be built to handle include (but are not limited to): vector addition, vector multiplication, dot product of two vectors, and vector comparison (minimum and maximum).

[0023] Another important feature of a SIMD engine is the ability to handle different levels of precision. For example, a SIMD engine with an input vector width of 128 bits can interpret the input vector as: 1 scalar of 128 bits, 2 scalars of 64 bits each, 4 scalars of 32 bits each, 8 scalars of 16 bits each, or 16 scalars of 8 bits each. Thus, a SIMD engine can operate in different "modes", such as 8-bit mode, 16-bit mode, or 32-bit mode.

[0024] Although SIMD engines are efficient in processing vectors and can execute a variety of instructions, they require a large amount of control logic and local memory. It can be seen that in order to obtain the maximum number of operations per unit silicon area, the number of SIMD engines needs to be maximized. One way to achieve this is to have a design in which multiple SIMD engines can share control logic and memory resources. However, this imposes restrictions on the types of operations that SIMD engines can perform, and additional logic may be required to drive the SIMD engines. Therefore, it will be necessary to have an improved computer architecture that includes SIMD engines. Summary of the invention

[0025] Therefore, the present invention provides an improved computer architecture, which includes multiple single instruction multiple data (SIMD) engines operating in parallel. The operand A register file stores a first group of one or more operand values ​​(operand A values), wherein each of the operand A values ​​includes multiple operand numbers. The operand B register file stores a second plurality of one or more operand values ​​(operand B values), wherein each of the operand B values ​​includes multiple operand numbers. In a specific embodiment, each of the operand A values ​​and the operand B values ​​includes four 32-bit operand numbers. In another embodiment, each of the operand A values ​​and the operand B values ​​includes eight 16-bit operand numbers.

[0026] An input distribution block includes an operand A distribution circuit and an operand B distribution circuit.

[0027] The operand A distribution circuit is coupled to receive operand A values ​​from the operand A register file. The operand A distribution circuit selectively routes one or more of the operands of the received operand A values ​​to create a plurality of input operand A values, wherein each of the input operand A values ​​is routed to a corresponding one of the plurality of SIMD engines.

[0028] In one mode, the operand A distribution circuit is controlled to route the received operand A value to each of the SIMD engines in parallel. For example, if the received operand A value includes four operand numbers [w, x, y, z], and there are four parallel SIMD engines, each of the SIMD engines will receive the four operand numbers [w, x, y, z].

[0029] In another mode, the operand A distribution circuit is controlled to route a single one of the operand numbers of the received operand A value to each of the SIMD engines in parallel. In this mode, a selected single one of the operand numbers is repeated multiple times to create each of the input operand A values. For example, if the received operand A value includes four operand numbers [w, x, y, z], operand number [w] is selected, and there are four parallel SIMD engines, then each of the SIMD engines will receive an input operand A value including the four operand numbers [w, w, w, w].

[0030] In yet another mode, the operand A distribution circuit is controlled to route the different operand numbers of the received operand A value to each of the SIMD engines in parallel. In this mode, each different operand number is repeated multiple times to create each of the input operand A values. For example, if the received operand A value includes four operand numbers [w, x, y, z] and there are four parallel SIMD engines, then one of the four SIMD engines will receive the input operand A value [w, w, w, w], one of the four SIMD engines will receive the input operand A value [x, x, x, x], one of the four SIMD engines will receive the input operand A value [y, y, y, y], and one of the four SIMD engines will receive the input operand A value [z, z, z, z].

[0031] The operand B distribution circuit is coupled to receive one or more operand B values ​​from the operand B register file. The operand B distribution circuit selectively routes one or more of the operands from one or more of the received operand B values ​​to create a plurality of input operand B values, wherein each of the input operand B values ​​is routed to a corresponding one of the plurality of SIMD engines.

[0032] In one mode, the operand B distribution circuit is controlled to route the received operand B value to each of the SIMD engines in parallel. For example, if the received operand B value includes four operand numbers [a, b, c, d], and there are four parallel SIMD engines, each of the SIMD engines will receive the four operand numbers [a, b, c, d].

[0033] In another mode, the operand B allocation circuit includes multiple buffers to store multiple operand B values. The operand B selection logic is used to select which of the SIMD engines receives which of the buffered operand B values. For example, if the buffered operand B values ​​include [a, b, c, d], [e, f, g, h], [i, j, k, l], and [m, n, o, p], and there are four parallel SIMD engines, then one of the four SIMD engines can receive input operand B value [a, b, c, d], one of the four SIMD engines can receive input operand B value [e, f, g, h], one of the four SIMD engines can receive input operand B value [i, j, k, l], and one of the four SIMD engines can receive input operand B value and [m, n, o, p]. Alternatively, two of the four SIMD engines may receive input operand B values ​​[a, b, c, d], one of the four SIMD engines may receive input operand B values ​​[e, f, g, h], and one of the four SIMD engines may receive input operand B values ​​[i, j, k, l].

[0034] In various embodiments, the operand B register file may include a single register file (so that multiple operand B values ​​are loaded into the operand B buffer in serial fashion) or multiple register files (so that multiple operand B values ​​are loaded into the operand B buffer in parallel). If the operand B register file is implemented using multiple register files, the operand B buffer may be implemented using a double buffer configuration, where the operand B values ​​are passed from the operand B register file to the operand B allocation circuitry while the operand B values ​​are passed from the operand B allocation circuitry to the SIMD engine.

[0035] In yet another mode, the operand B distribution circuit receives multiple operand B values ​​in parallel from the operand B register file. These received operand B values ​​are provided to the shift logic circuit within the operand B distribution circuit. The control logic specifies the shift amount (in units of operands) that the shift logic circuit introduces to the received operand B values. The shifted operand B values ​​are buffered in the operand B distribution circuit and then routed to the SIMD engine in parallel.

[0036] The improved computer architecture also includes a plurality of output register sets, each output register set being connected to a corresponding one of the plurality of SIMD engines. Data (e.g., dot product values) are selectively transferred between the output register sets and the corresponding SIMD engines to implement operations such as multiplication-accumulation operations. In one embodiment, each of the plurality of output register sets is independently addressed to provide flexibility for the operations performed.

[0037] In certain embodiments, the computer architecture of the present invention enables efficient sparse matrix multiplication.

[0038] The invention will be more fully understood in view of the following description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a block diagram of a conventional multiplier-accumulator (MAC) unit.

[0040] Figure 2 is a block diagram of a conventional system including multiple parallel MAC units for computing multiple dot products in parallel.

[0041] Figure 3 It is through Figure 2 Various elements of multiple parallel MAC units are grouped to create a block diagram of a conventional single instruction multiple data (SIMD) engine.

[0042] Figure 4 is a block diagram of a computer system according to one embodiment of the present invention.

[0043] Figure 5 is a block diagram illustrating an architecture (Architecture 1A) for routing a first operand value (Operand A) having four 32-bit operands from an Operand A register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0044] Figure 6 is a block diagram illustrating an architecture (Architecture 1A) for routing a first operand value (Operand A) having eight 16-bit operands from an Operand A register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0045] Figure 7 and Figure 8 is a block diagram illustrating an architecture (Architecture 2A) for routing a first operand value (Operand A) having four 32-bit operands from an Operand A register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0046] Fig. 9 and Fig.10 is a block diagram illustrating an architecture (Architecture 2A) for routing a first operand value (Operand A) having eight 16-bit operands from an Operand A register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0047] Fig.11 is a block diagram illustrating an architecture (Architecture 3A) for routing a first operand value (Operand A) having four 32-bit operands from an Operand A register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0048] Fig.12and Fig.13 is a block diagram illustrating an architecture (architecture 3A) for routing a first operand value (operand A) having eight 16-bit operands from an operand A register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0049] Fig.14 is a block diagram illustrating an architecture (Architecture 1B) for routing a second operand value (Operand B) having four 32-bit operands from an operand B register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0050] Fig.15 is a block diagram illustrating an architecture (Architecture 1B) for routing a second operand value (Operand B) having eight 16-bit operands from an operand B register file to multiple SIMD engines in accordance with one embodiment of the present invention.

[0051] Fig.16 is a block diagram illustrating an architecture (Architecture 2B) for routing a second operand value (Operand B) having four 32-bit operands from an operand B register file and a plurality of operand B buffers to a plurality of SIMD engines in accordance with one embodiment of the present invention.

[0052] Fig.17 is a block diagram illustrating an architecture (Architecture 3B) for routing a second operand value (Operand B) having four 32-bit operand numbers from multiple parallel operand B register files and multiple operand B buffers to multiple SIMD engines according to one embodiment of the present invention.

[0053] Fig.18 is a block diagram illustrating an architecture (Architecture 3B) for routing a second operand value (Operand B) having four 32-bit operand numbers from multiple parallel operand B register files and multiple operand B double buffers to multiple SIMD engines according to one embodiment of the present invention.

[0054] Fig.19 is a block diagram illustrating an architecture (Architecture 4B) for routing a second operand value (Operand B) having four 32-bit operand numbers from multiple parallel operand B register files, shift logic circuits, and multiple operand B buffers to multiple SIMD engines in accordance with one embodiment of the present invention.

[0055] Fig. 20 is a block diagram of a computer system including a SIMD block having four parallel SIMD engines and an output circuit having four parallel output register sets according to one embodiment of the present invention.

[0056] Fig.21 yes Fig. 20 A block diagram of a computer system illustrating addressing of four parallel output register sets according to one embodiment of the present invention.

[0057] Fig. 22 is a unified structure according to an embodiment of the present invention. Figures 4 to 21 Block diagrams of hardware systems specifying various architectures and features of different operands and outputs.

[0058] Fig.23 is a block diagram of a computer architecture for performing matrix multiplication according to one embodiment of the present invention.

[0059] Fig.24 is a diagram of an embodiment according to the present invention to be Fig.23 Diagram of the computer architecture for multiplying two matrices I and J.

[0060] Fig.25 is a diagram illustrating an embodiment of the present invention. Fig.24 A block diagram of the way in which the contents of matrix I and matrix J are logically stored within the system memory.

[0061] Fig.26 is a diagram illustrating an embodiment of the present invention. Fig.24 The result of the multiplication of matrix I and matrix J is stored in Fig.23 A block diagram of the way the output registers are organized within a computer architecture.

[0062] Fig. 27 and Fig.28 is a block diagram of a computer architecture during various stages of sparse matrix multiplication according to one embodiment of the present invention.

[0063] Fig.29 , Fig.30 and Fig.31 is a block diagram of a computer architecture during various stages of sparse matrix multiplication according to an alternative embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following subsections describe various efficient SIMD engine architectures. Specifically, a method for operating multiple SIMD engines in parallel is proposed, and a method for supplying input to the SIMD engines is described. Although the following description uses an example of implementing a 128-bit width input operand and 4 SIMD engines, it is noted that the described examples can be extended to smaller or larger input operand widths and / or fewer or more SIMD engines.

[0065] Figure 44 is a block diagram of a computer system 400 including various hardware resources required for operating a SIMD block 401 according to one embodiment. These resources include an operand buffer 410 (which includes an operand A register file 411 and an operand B register file 412), an input allocation block 415 (which includes an operand A allocation circuit 416 and an operand B allocation circuit 417), a SIMD block 401, an output circuit 420, a control logic 430 (which includes a state machine and scheduler 431, a control register 432 and an operand packaging circuit 433) and a system memory 440. The important parameters for SIMD operations are the operand, the type of operation and the address of the output circuit. These parameters will be described in more detail in subsequent sections.

[0066] In general, the control logic 430 controls the writing of operand values ​​(e.g., matrix entries) into the operand A register file 411 and the operand B register file 412. More specifically, the state machine and scheduler 431 causes the operand packing circuit 433 to retrieve the matrix entries stored in the system memory 440. The operand packing circuit 433 packs these matrix entries to form operand values ​​according to the operation to be performed. In one embodiment, various operations are defined by entries in the control register 432. As described in more detail below, some operations (e.g., sparse matrix multiplication) require that matrix entries with zero values ​​be omitted from the operand values ​​provided to the operand buffer 410. The state machine and scheduler 431 controls the writing of the operand values ​​provided by the operand packing circuit 433 into the operand A register file 411 and the operand B register file 412. The state machine and scheduler 431 also controls the reading of operand values ​​from the operand A register file 411 and the operand B register file 412 , where these read values ​​are provided to the operand A allocation circuit 416 and the operand B allocation circuit 417 within the input allocation block 415 .

[0067] In general, state machine and scheduler 431 provides addresses to input distribution block 415, where these addresses control the manner in which operand A distribution circuit 416 routes operand A values ​​received from operand A register file 411 to SIMD block 401, and also controls the manner in which operand B distribution circuit 417 routes operand B values ​​received from operand B register file 412 to SIMD block 401. As described in more detail below, operand B distribution circuit 417 may include a buffer for storing a plurality of operand B values, and shift logic that controls the amount of shift to be applied to operand B values ​​received from operand B register file 412.

[0068] The state machine and scheduler 431 also provides addresses for accessing memory banks included within the output circuit 420. These addresses include: read addresses that enable accumulated values ​​stored in the memory banks to be routed to the SIMD block 401 for multiply-accumulate operations; and write addresses that enable updated accumulated values ​​provided by the SIMD block 401 to be written back to the memory banks within the output circuit 420.

[0069] Control registers 432 store values ​​that control the manner in which state machine and scheduler 431 generates various addresses for different operating modes (which are described in more detail below). The operation of various elements of computer system 400 is described in more detail below for various modes (ie, architectures).

[0070] Architecture of operand A

[0071] Three architectures are proposed for input operand A, which are described in more detail below. In the following examples, SIMD block 401 includes four SIMD engines operating in parallel, where each of the four SIMD engines receives a corresponding input operand A having a width of 128 bits. A single entry from operand A register file 411 (which is included in operand buffer 410) is 128 bits. This entry is hereinafter referred to as a register file word. In the described embodiment, each of the four SIMD engines within SIMD block 401 is associated with Figure 3 The SIMD engine 300 is the same.

[0072] Architecture 1A

[0073] In a first architecture (architecture 1A) for providing operand A to SIMD block 401, each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401 receives a complete register file word (which includes four 32-bit word values ​​w, x, y and z) as input operand A.

[0074] Figure 54 is a block diagram illustrating an embodiment of a first operand A architecture (architecture 1A) in which four 32-bit input words [w, x, y, z] stored in operand A register file 411 are routed to operand A allocation circuit 416. In one embodiment, operand A allocation circuit 416 includes a buffer that stores received input words [w, x, y, z]. Operand A allocation circuit 416 also includes circuitry for performing a switching / demultiplexing function, wherein the buffered input words [w, x, y, z] are routed in parallel to each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. Therefore, in architecture 1A, each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) receives a complete register file word [w, x, y, z] as input operand A.

[0075] Figure 6 4 is a block diagram illustrating another embodiment of a first operand A architecture (architecture 1A) in which eight 16-bit input words [s, t, u, v, w, x, y, z] stored in operand A register file 411 are routed to operand A allocation circuit 416. These input words [s, t, u, v, w, x, y, z] are buffered within operand A allocation circuit 416 and then routed in parallel to each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. Likewise, in architecture 1A, each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) receives a complete register file word [s, t, u, v, w, x, y, z] as input operand A.

[0076] Architecture 2A

[0077] In a second architecture (architecture 2A) for providing operand A to SIMD block 401, each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401 receives a single input word from operand A register file 411, where the single input word is repeated multiple times to match the input width of operand A. Input distribution block 415 selects the single input word by specifying the index of the single input word to be spread within operand A register file 411.

[0078] Figure 74 is a block diagram illustrating an embodiment of a second operand A architecture (architecture 2A), wherein four 32-bit input words [w, x, y, z] stored in operand A register file 411 are routed to operand A allocation circuit 416. As described above, operand A allocation circuit 416 includes a buffer for storing received input words [w, x, y, z]. State machine and scheduler 431 provides an index value to operand A allocation circuit 416, wherein the index value specifies input word [w]. In response, operand A allocation circuit 416 performs a switching / demultiplexing operation, wherein input word [w] is routed in parallel to each of four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. That is, the 32-bit input word [w] is effectively repeated four times to provide a 128-bit input operand A, which is composed of the 32-bit word [w] repeated four times. The 128-bit input operand A[w, w, w, w] is provided in parallel to each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3).

[0079] Figure 8 4 is a block diagram illustrating another example of a second operand A architecture (architecture 2A) in which a state machine and scheduler 431 provides an index value specifying an input word y to operand A allocation circuit 416. In response, operand A allocation circuit 416 performs a switching / demultiplexing operation in which the input word [y] is routed in parallel to each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. That is, the 32-bit input word [y] is effectively repeated four times to provide a 128-bit input operand A [y, y, y, y], which is provided in parallel to each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3).

[0080] Fig. 94 is a block diagram illustrating another embodiment of a second operand A architecture (architecture 2A), wherein eight 16-bit input words [s, t, u, v, w, x, y, z] from operand A register file 411 are buffered in operand A allocation circuit 416. State machine and scheduler 431 provides an index value to operand A allocation circuit 416, wherein the index value specifies input word [u]. In response, operand A allocation circuit 416 performs a switching / demultiplexing operation, wherein input word [u] is routed in parallel to each of four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. That is, the 16-bit input word [u] is effectively repeated eight times to provide a 128-bit input operand A [u, u, u, u, u, u, u, u], which is provided in parallel to each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3).

[0081] Fig.10 4 is a block diagram illustrating another example of a second operand A architecture (architecture 2A), wherein eight 16-bit input words [s, t, u, v, w, x, y, z] from operand A register file 411 are buffered in operand A allocation circuit 416. State machine and scheduler 431 provides an index value to operand A allocation circuit 416, wherein the index value specifies input word [z]. In response, operand A allocation circuit 416 performs a switching / demultiplexing operation, wherein the input word [z] is routed in parallel to each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. That is, the 16-bit input word [z] is effectively repeated eight times to provide a 128-bit input operand A [z, z, z, z, z, z, z, z], which is provided in parallel to each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3).

[0082] Architecture 3A

[0083] In the third architecture (architecture 3A) for providing input operand A to SIMD block 401, each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401 receives a single input word from operand A register file 411, where the single input word is repeated multiple times to match the input width of operand A. However, different SIMD engines are provided with different input words. In one embodiment, the input words are assigned to the SIMD engines in a round-robin manner. The input allocation block 415 selects a single input word for each SIMD by specifying the index of each input word to be provided to each SIMD.

[0084] Fig.11 4 is a block diagram illustrating one embodiment of a third operand A architecture (architecture 3A) in which four 32-bit input words [w, x, y, z] stored in operand A register file 411 are received by (and buffered in) operand A allocation circuit 416. State machine and scheduler 431 provides index values ​​to operand A allocation circuit 416, where the index values ​​specify which of the input words [w, x, y, z] will be provided to the SIMD engine. In this example, the index values ​​specify that input words w, x, y, and z will be provided to SIMD0, SIMD1, SIMD2, and SIMD3, respectively. In response to the index value, the operand A allocation circuit 416 performs a switching / demultiplexing operation in which: the 32-bit input word [w] is repeated four times to create a 128-bit operand A value [w, w, w, w], which is routed to SIMD0; the 32-bit input word [x] is repeated four times to create a 128-bit operand A value [x, x, x, x], which is routed to SIMD1; the 32-bit input word [y] is repeated four times to create a 128-bit operand A value [y, y, y, y], which is routed to SIMD2; and the 32-bit input word [z] is repeated four times to create a 128-bit operand A value [z, z, z, z], which is routed to SIMD3. In the above manner, the four 32-bit entries [w, x, y, z] of the 128-bit register file word are assigned to the four SIMD engines in a round-robin manner (ie, SIMD0 is assigned w, SIMD1 is assigned x, SIMD2 is assigned y, and SIMD3 is assigned z).

[0085] Fig.124 is a block diagram illustrating another embodiment of a third operand A architecture (architecture 3A) in which eight 16-bit input words [s, t, u, v, w, x, y, z] stored in operand A register file 411 are received by (and buffered in) operand A allocation circuit 416. State machine and scheduler 431 provides index values ​​to operand A allocation circuit 416, wherein the index values ​​specify which of the input words [s, t, u, v, w, x, y, z] are to be provided to the SIMD engine. In this example, the index values ​​specify that input words s, t, u, and v are to be provided to SIMD0, SIMD1, SIMD2, and SIMD3, respectively. In response to the index value, operand A allocation circuit 416 performs a switching / demultiplexing operation in which: the 16-bit input word [s] is repeated eight times to create a 128-bit operand A value [s, s, s, s, s, s, s], which is routed to SIMD0; the 16-bit input word [t] is repeated eight times to create a 128-bit operand A value [t, t, t, t, t, t, t], which is routed to SIMD1; the 16-bit input word [u] is repeated eight times to create a 128-bit operand A value [u, u, u, u, u, u, u, u], which is routed to SIMD2; and the 16-bit input word [v] is repeated eight times to create a 128-bit operand A value [v, v, v, v, v, v, v, v], which is routed to SIMD3. In the above manner, the four 16-bit entries [s, t, u, v] of the 128-bit register file word are distributed to the four SIMD engines in a round-robin manner.

[0086] Note that Fig.12 In the 16-bit input pattern represented by , there are more entries in the register file word (8) than there are SIMD engines (4). Therefore, the same register file word can be used twice: once to distribute the first four entries to the four SIMD engines (e.g. Fig.12 ), and another time to distribute the last four entries to the four SIMD engines (as shown Fig.13 In one embodiment, the index value provided by the state machine and scheduler 431 specifies which four entries of the register file word are allocated to provide the input operand A.

[0087] Fig.13 It is a graphic illustration Fig.12A block diagram of a continuation of the beginning allocation, where the 16-bit input word [w] is repeated eight times to create a 128-bit operand A value [w, w, w, w, w, w, w, w], which is routed to SIMD0; the 16-bit input word [x] is repeated eight times to create a 128-bit operand A value [x, x, x, x, x, x, x, x], which is routed to SIMD1; the 16-bit input word [y] is repeated eight times to create a 128-bit operand A value [y, y, y, y, y, y, y], which is routed to SIMD2; and the 16-bit input word [z] is repeated eight times to create a 128-bit operand A value [z, z, z, z, z, z, z, z], which is routed to SIMD3.

[0088] Combine the following Fig. 22 More details are provided regarding the routing of input operand A according to architectures 1A, 2A, and 3A.

[0089] Note that the foregoing description of architectures 1A, 2A, and 3A implements a 32-bit input mode and a 16-bit input mode. However, these embodiments are provided for illustrative purposes only. These ideas are general and can be extended to other input modes (e.g., 8-bit input modes) in a straightforward manner. In addition, while architectures 1A, 2A, and 3A have been described in conjunction with embodiments including 4 SIMD engines and 128-bit register file words, other numbers of SIMD engines and register file word widths can be used in a straightforward manner in other embodiments.

[0090] In actual hardware implementation, multiple architectures can be implemented together by sharing hardware resources. The hardware can be programmed to operate different architectures into modes that can be selected by some register settings. For example, control register 432 ( Figure 4 ) may store values ​​that cause state machine and scheduler 431 to implement architecture 1A, architecture 2A, or architecture 3A for operand A in the manner described above. That is, the switching and demultiplexing logic implemented in operand A allocation circuitry 416 may be driven by the settings stored by control register 432.

[0091] For example, in architecture 2A, the index of the single value to be spread needs to be provided to operand A allocation circuit 416. This index may have different interpretations depending on whether the data is 8, 16, or 32 bits wide (which may be specified by a control register setting).

[0092] Similarly, in architecture 3A, the index of the value to be spread to SIMD0 needs to be provided to the operand A allocation circuit 416. Based on this index, the index of the value to be spread to the other SIMD engines (SIMD1, SIMD2 and SIMD3) can be inferred by hardware by incrementing. Or all four indices can be provided to the operand A allocation circuit 416.

[0093] The data stored in the buffer of operand A allocation circuit 416 can be reused in multiple cycles, thereby not needing to read register file words from operand A register file 411 every cycle. Separate control logic can supply the flag that specifies which cycle needs to load data from operand A register file 411. In addition, operand A allocation circuit 416 can contain multiple buffers to hold register file word data, wherein control logic specifies the buffer index for writing and reading. In one embodiment, operand A allocation circuit 416 contains two buffers: one for writing, and one for reading, which are used in a ping-pong manner. State machine and scheduler 431 automatically manage read and write indexes. This scheme is generally referred to as double buffering. In this case, no additional control logic is needed to specify the buffer index for reading and writing.

[0094] The architecture of operand B

[0095] According to other embodiments, multiple architectures are used to provide input operand B to SIMD block 401. As described below, operand B allocation circuit 417 can be configured in four different architectures (Architecture 1B, Architecture 2B, Architecture 3B, and Architecture 4B) to provide input operand B to SIMD block 401.

[0096] Architecture 1B

[0097] In a first architecture (architecture 1B) for providing operand B to SIMD block 401, each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401 receives a complete register file word (which includes four 32-bit word values ​​a, b, c, and d) as input operand B. Note that architecture 1B for providing operand B to SIMD block 401 is similar to architecture 1A for providing operand A to SIMD block 401.

[0098] Fig.144 is a block diagram illustrating an embodiment of a first operand B architecture (architecture 1B) in which four 32-bit input words [a, b, c, d] stored in operand B register file 412 are routed to operand B allocation circuit 417. In one embodiment, operand B allocation circuit 417 includes a buffer to store received input words [a, b, c, d]. Operand B allocation circuit 417 also includes circuitry for performing a switching / demultiplexing function, in which the buffered input words [a, b, c, d] are routed in parallel to each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. Therefore, in architecture 1B, each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) receives a complete register file word [a, b, c, d] as an input operand B.

[0099] Fig.15 4 is a block diagram illustrating another embodiment of a first operand B architecture (architecture 1B) in which eight 16-bit input words [a, b, c, d, e, f, g, h] stored in operand B register file 412 are routed to operand B allocation circuit 417. These input words [a, b, c, d, e, f, g, h] are buffered within operand B allocation circuit 417 and then routed in parallel to each of the four SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) included in SIMD block 401. Likewise, in architecture 1B, each of the SIMD engines (SIMD0, SIMD1, SIMD2, SIMD3) receives a complete register file word [a, b, c, d, e, f, g, h] as input operand B.

[0100] Architecture 2B

[0101] In the architecture currently considered (for operand A and for operand B), all SIMD engines use data from a single register file word at a given time. This can make the architecture rigid in terms of the types of operations they can support. According to one embodiment, different SIMD engines are provided with different register file words than the operand register file.

[0102] One way to accomplish this is to allow multiple reads of the register file to occur simultaneously. While this is possible, the hardware complexity may be prohibitive.

[0103] In one embodiment, multiple entries (not necessarily different) can be read simultaneously from operand B register file 412. The most general way to implement this is to use a multi-read port memory to implement the register file 412. A memory with four read ports can be used to read four entries from operand B register file 412 simultaneously. However, such a memory configuration has high hardware complexity (occupying a relatively large area and consuming relatively high power). Therefore, a preferred embodiment of the present invention includes a low complexity method and structure for supplying (possibly) different input operand B values ​​to different SIMD engines. Although these preferred embodiments may not provide as much universality as a wide range of (multiple read ports) methods, they are efficient for the purpose of the algorithm to be implemented.

[0104] According to one embodiment, a second architecture (architecture 2B) for providing input operand B to the SIMD engines is provided, in which a small number of entries from the operand B register file 412 are buffered in the operand B distribution circuit 417 and then distributed to the SIMD engines of the SIMD block 401. Intuitively, this can be thought of as a way to give some flexibility to each SIMD by allowing them to address any entry from the small number of entries. This keeps the hardware complexity small.

[0105] The main features of the second architecture (architecture 2B) for providing input operand B to SIMD block 401 can be defined as follows. Operand B allocation circuit 417 includes multiple operand B buffers to hold values ​​read from operand B register file 412. Each of these operand B buffers can hold a complete register file word. Each SIMD can receive a register file word stored in any one of the operand B buffers. The buffer selection mechanism is used to specify which of the operand B buffers is connected to each of the SIMD engines. The operand B buffers are filled one at a time from the operand B register file 412. When a new register file word needs to be loaded from the operand B register file 412 into the operand B buffer, one of the previous operand B buffers is overwritten. There can be multiple schemes to determine which of the operand B buffers needs to be overwritten. A simple scheme is to overwrite the operand B buffer with the oldest data (that is, the operand B buffer is used in a circular manner). In another scheme, control logic 430 can specify which operand B buffer needs to be overwritten.

[0106] It is not necessary to load data from operand B register file 412 into operand B buffer during every cycle. Separate control logic 430 may specify a flag for each cycle to indicate whether new data needs to be read from operand B register file 412 into operand B buffer of operand B allocation circuit 417. In an actual hardware implementation, each operand B buffer may use a double buffering scheme so that read and write operations to the operand B buffer do not occur in the same cycle.

[0107] Fig.16 4 is a block diagram of a second architecture (architecture 2B) for providing input operands B to a SIMD engine according to one embodiment. In the illustrated example, the operand B allocation logic 417 includes four operand B buffers B0-B3, each of which is capable of storing a complete register word from the operand B register file 412. Although four operand B buffers are shown, it will be understood that the number of operand buffers can be any number greater than 1. In the illustrated example, the operand B buffers B0-B3 store register file words received from the operand B register file 412. More specifically, the operand B buffers B0, B1, B2, and B3 store the values ​​[a, b, c, d], [e, f, g, h], [i, j, k, l], and [m, n, o, p], respectively (where each of the values ​​ap is a 32-bit word).

[0108] Operand B buffer selection logic 1601 (which may be included in the state machine and scheduler 431 of control logic 430) is used to determine the manner in which the contents of operand B buffers B0, B1, B2, and B3 are provided to SIMD0, SIMD1, SIMD2, and SIMD3. In one embodiment, operand B buffer selection logic 1601 includes four buffer selection entries bs0, bs1, bs2, and bs3, which store which of the operand B buffers B0-B3 specifies which of SIMD0, SIMD1, SIMD2, and SIMD3 provide their contents to, respectively. In the illustrated example, four entries bs0, bs1, bs2, and bs3 specify operand B buffers B0, B0, B1, and B2, respectively, indicating that the contents of operand B buffer B0 (i.e., [a, b, c, d]) are provided to SIMD0 and SIMD1, the contents of operand B buffer B1 (i.e., [e, f, g, h]) are provided to SIMD2, and the contents of operand B buffer B2 (i.e., [i, j, k, l]) are provided to SIMD3. In subsequent cycles, the buffer selection can be changed by changing the buffer selection entries bs0, bs1, bs2, and bs3. Note that if the number of operand buffers is reduced to 1, then architecture 2B will be equivalent to architecture 1B.

[0109] Architecture 3B

[0110] Another approach to effectively allow multiple reads from operand B register file 412 is to implement operand B register file 412 using multiple register files, each of which allows a single read operation to be performed at a time. As previously mentioned, having a large memory with 4 read ports is more expensive than four small memories with one read port. However, it is worth noting that a larger memory with 4 read ports provides greater flexibility in the data that can be read. When using four smaller memories with a single read port, four entries can be read at a given time, but each of the entries must belong to a different memory. For a 4-read port memory that allows any 4 entries to be read simultaneously, there is no problem.

[0111] The main features of the third architecture (architecture 3B) for providing input operand B to SIMD block 401 can be defined as follows. There is more than one register file for operand B. In one particular case, the number of register files for operand B is equal to the number of SIMD engines included in SIMD block 401. Therefore, if there are four SIMD engines, there will be four corresponding operand B register files. However, other cases are possible, and it is easy to extend the architecture to these cases.

[0112] Multiple operand B register files may be read simultaneously. In the simple case, each SIMD receives its input operand B directly from one of the operand B register files. If the number of SIMD engines is equal to the number of operand B register files, each of the SIMD engines may receive input operand B from a corresponding one of the operand B register files.

[0113] In general, operand B allocation circuitry 417 may contain an operand buffer (similar to architecture 2B) to hold data read from the operand B register file. This may allow multiple cycles to use the same data. In addition, due to the reuse of buffered data, the operand B register file does not need to be read every cycle. The load flag may specify the cycle in which data needs to be read from the operand B register file to the operand B allocation circuitry 417. A separate block may also specify the address of the buffer to be loaded for each SIMD, as described above in conjunction with architecture 2B.

[0114] Fig.174B is a block diagram of a third architecture (architecture 3B) for providing input operands B to SIMD block 401 according to one embodiment. In the illustrated example, four operand B register files 4120, 4121, 4122, and 4133 provide four corresponding operand B register words (e.g., [a0, b0, c0, d0] [e0, f0, g0, h0] [i0, j0, k0, l0] and [m0, n0, o0, p0]) to operand B allocation circuit 417. In the illustrated example, operand B allocation circuit 417 routes the register file words provided by operand B register files 4120, 4121, 4122, and 4133 to buffer memories BM0, BM1, BM2, and BM3 within operand B allocation circuit 417, respectively.

[0115] Operand B buffer selection logic 1701 (which may be included in the state machine and scheduler 431 of control logic 430) is used to determine the manner in which the contents of operand B buffers BM0, BM1, BM2, and BM3 are provided to SIMD0, SIMD1, SIMD2, and SIMD3. In one embodiment, operand B buffer selection logic 1701 includes four buffer selection entries bms0, bms1, bms2, and bms3, which store values ​​specifying which of operand B buffers BM0-BM3 provide their contents to SIMD0, SIMD1, SIMD2, and SIMD3, respectively. In the illustrated example, four entries bms0, bms1, bms2, and bms3 specify operand B buffers BM0, BM1, BM2, and BM3, respectively, indicating that the contents of operand B buffer BM0 (i.e., [a0, b0, c0, d0]) are provided to SIMD0, the contents of operand B buffer BM1 (i.e., [e0, f0, g0, h0]) are provided to SIMD1, the contents of operand B buffer BM2 (i.e., [i0, j0, k0, l0]) are provided to SIMD2, and the contents of operand B buffer BM3 (i.e., [m0, n0, o0, p0]) are provided to SIMD3. In subsequent cycles, the buffer selection can be changed by changing the buffer memory selection entries bms0, bms1, bms2, and bms3.

[0116] Fig.18 is a block diagram of a third architecture (Architecture 3B) for providing input operand B to SIMD block 401 according to an alternative embodiment. Fig.18In the embodiment of the present invention, operand B allocation circuit 417 includes dual operand B buffers B01-B02, B11-B12, B21-B22 and B31-B32, which store data provided by operand B register files 4120, 4121, 4122 and 4133, respectively. In the illustrated example, operand B register file words [a0, b0, c0, d0] and [a1, b1, c1, d1] from operand B register file 4120 are stored in operand B buffers B02 and B01, respectively. Operand B register file words [e0, f0, g0, h0] and [e1, f1, g1, h1] from operand B register file 4121 are stored in operand B buffers B12 and B11, respectively. Operand B register file words [i0, j0, k0, l0] and [i1, j1, k1, l1] from operand B register file 4122 are stored in operand B buffers B22 and B21, respectively. Operand B register file words [m0, n0, o0, p0] and [m1, n1, o1, p1] from operand B register file 4123 are stored in operand B buffers B32 and B31, respectively.

[0117] Operand B buffer selection logic 1801 (which may be included in the state machine and scheduler 431 of control logic 430) is used to determine the manner in which the contents of operand B buffers B01-B02, B11-B12, B21-B22, and B31-B32 are provided to SIMD0, SIMD1, SIMD2, and SIMD3. In one embodiment, operand B buffer selection logic 1801 includes four buffer selection entries bs01, bs11, bs21, and bs31, which store values ​​specifying which of operand B buffers B01-B02, B11-B12, B21-B22, and B31-B32 provide their contents to SIMD0, SIMD1, SIMD2, and SIMD3, respectively. In the illustrated example, four entries bs01, bs11, bs21, and bs31 specify operand B buffers B02, B12, B22, and B32, respectively, indicating that the contents of operand B buffer B02 (i.e., [a0, b0, c0, d0]) are provided to SIMD0, the contents of operand B buffer B12 (i.e., [e0, f0, g0, h0]) are provided to SIMD1, the contents of operand B buffer B22 (i.e., [i0, j0, k0, l0]) are provided to SIMD2, and the contents of operand B buffer B32 (i.e., [m0, n0, o0, p0]) are provided to SIMD3. In one embodiment, operand B allocation circuitry 417 includes switching / demultiplexing circuitry that performs the above-described routing in response to buffer select entries bs01, bs11, bs21, and bs31. Note that operand B buffer selection logic 1801 may select any of operand buffers B01-B02, B11-B12, B21-B22, and B31-B32 to provide input operand B to any of the SIMD engines. For example, buffer selection entry bs01 may store a value (B31) that causes the contents of operand B buffer B31 (i.e., [m1, n1, o1, p1]) to be routed to SIMD0. In subsequent cycles, the buffer selection may be changed by changing buffer selection entries bs01, bs11, bs21, and bs31. Note that in other embodiments, a different number of operand B buffers may be included in operand B allocation circuit 417.

[0118] Architecture 4B

[0119] In the fourth architecture (architecture 4B) for providing operand B to SIMD block 401, an architecture similar to architecture 3B is provided, with the added feature that each operand B register file allows two entries to be read at a time and selects one register file word value of data by applying some shift operations. Control logic 340 specifies the addresses of the two rows to be read from each operand B register file, and the shift amount to be applied to the entries read from these two rows. This function is usually implemented in hardware by implementing each operand B register file memory as two memory banks. This allows two entries to be read simultaneously. The two register file words are then fed to the shift logic module, which receives the shift amount as an input parameter and outputs one register file word value of data. The addresses and shift amounts of the two banks are supplied by the state machine and scheduler 431.

[0120] Fig.19 4B) for providing input operand B to SIMD block 401 according to one embodiment. In the illustrated example, there are four operand B register files 19120-19123, wherein each of these operand B register files includes two memory banks. Thus, operand B register files 19120, 19121, 19122, and 19123 include memory banks 1912, 19123, 19124, 19125, 19126, 19127, 19128, 19129, 19130, 19131, 19132, 19133, 19134, 19135, 19136, 19137, 19138, 19139, 19140, 19141, 19142, 19143, 00 -1912 01 , 1912 10 -1912 11 , 1912 20 -1912 21 and 1912 30 -1912 31 Memory Bank 1912 00 -1912 01 , 1912 10 -1912 11 , 1912 20 -1912 21 and 1912 30 -1912 31 Each of the memory banks 1912 stores a different register file word. 00 Store register file words [a0, b0, c0, d0], [a2, b2, c2, d2], and [a4, b4, c4, d4] and memory bank 1912 01 Storage register file words [a1, b1, c1, d1], [a3, b3, c3, d3], and [a5, b5, c5, d5].

[0121] Memory Bank 1912 10Store register file words [e0, f0, g0, h0], [e2, f2, g2, h2], and [e4, f4, g4, h4] and memory bank 1912 11 Storage register file words [e1, f1, g1, h1], [e3, f3, g3, h3], and [e5, f5, g5, h5].

[0122] Memory Bank 1912 20 Store register file words [i0, j0, k0, l0], [i2, j2, k2, l2], and [i4, j4, k4, l4] and memory bank 1912 21 Storage register file words [i1, j1, k1, l1], [i3, j3, k3, l3], and [i5, j5, k5, l5].

[0123] Memory Bank 1912 30 Store register file words [m0, n0, o0, p0], [m2, n2, o2, p2], and [m4, n4, o4, p4] and memory bank 1912 31 Storage register file words [m1, n1, o1, p1], [m3, n3, o3, p3], and [m5, n5, o5, p5].

[0124] From the memory bank to 1912 00 -1912 01 , 1912 10 -1912 11 , 1912 20 -1912 21 and 1912 30 -1912 31 The read register file word is provided to shift logic circuit 1901 in operand B distribution circuit 417. The output of shift logic circuit 1901 is provided to operand B buffers B0, B1, B2 and B3 in operand B distribution circuit 417.

[0125] The control logic 430 (and more specifically, the state machine and scheduler 431) controls the slave memory bank 1912 00 -1912 01 , 1912 10 -1912 11 , 1912 20 -1912 21 and 1912 30 -1912 31 Typically, control logic 430 causes register file words to be read from memory bank 1912 simultaneously. 00 -1912 01 , 191210 -1912 11 , 1912 20 -1912 21 and 1912 30 -1912 31 The address provided to each of the memory bank pair may be selected so that two different consecutive register file words are read from each of the memory banks to provide the register file words required to perform the shift operation. For example, register file words [a0, b0, c0, d0] and [a1, b1, c1, d1] may be read from memory banks 1912, respectively. 00 and 1912 01 Read simultaneously; register file words [e0, f0, g0, h0] and [e1, f1, g1, h1] can be read from memory bank 1912 respectively 10 and 1912 11 Read simultaneously; register file words [i0, j0, k0, l0] and [i1, j1, k1, l1] can be read from memory bank 1912 respectively 20 and 1912 21 and register file words [m0, n0, o0, p0] and [m1, n1, o1, p1] can be read from memory bank 1912 respectively; 30 and 1912 31 Shift logic circuit 1901 receives eight register file words provided by operand B register file 19120-19123.

[0126] Control logic 340 also controls the amount of shift introduced by shift logic circuit 1901. In general, Table 1 below defines the values ​​provided to operand buffers B0-B3 in this example by shift logic circuit 1901 for various shift values. Note that each shift value introduces an additional 32 bits of shift to the received register file word pair. Fig.19 The diagram illustrates the result of a shift value of 1.

[0127] Table 1

[0128] Shift B0 B1 B2 B3 0 [a0 b0 c0 d0] [e0 f0 g0 h0] [i0 j0 k0 l0] [m0 n0 o0 p0] 1 [b0 c0 d0 a1] [f0 g0 h0 e1] [j0 k0 l0 i1] [n0 o0 p0 m1] 2 [c0 d0 a1 b1] [g0 h0 e1 f1] [k0 l0 i1 j1] [o0 p0 m1 n1] 3 [d0 a1 b1 c1] [h0 e1 f1 g1] [l0 i1 j1 k1] [p0 m1 n1 o1] 4 [a1 b1 c1 d1] [e1 f1 g1 h1] [i1 j1 k1 l1] [m1 n1 o1 p1]

[0129] The contents of operand B buffers B0, B1, B2 and B3 are routed to SIMD0, SIMD1, SIMD2 and SIMD3 respectively as input operand B. In the manner described above, shifts may be efficiently performed within the register file words stored by operand B register files 19120-19123.

[0130] Likewise, in actual hardware implementations, multiple architectures can be implemented together by sharing hardware resources. The hardware can be programmed to operate different architectures into modes that can be selected by registers.

[0131] Combine the following Fig. 22 More details are provided regarding the routing of input operand B according to architectures 1B, 2B, 3B, and 4B.

[0132] Although the foregoing description of architectures 1B, 2B, 3B, and 4B implements 32-bit input modes and 16-bit input modes, it will be appreciated that these architectures can be readily modified to implement input modes of other widths (e.g., 8-bit input modes). In addition, although architectures 1B, 2B, 3B, and 4B have been described in conjunction with embodiments including 4 SIMD engines and 128-bit register file words, other numbers of SIMD engines and register file word widths can be used in other embodiments in a straightforward manner.

[0133] In actual hardware implementations, multiple architectures can be implemented together by sharing hardware resources. The hardware can be programmed to operate different architectures into modes that can be selected by some register settings. For example, control register 432 ( Figure 4 ) may store a value that configures operand B allocation circuitry 415 to implement an architecture 1B, 2B, 3B, or 4B for operand B in the manner described above.

[0134] Output circuit system

[0135] Output circuit 420 ( Figure 4 ) is used to store the output of the SIMD engines (SIMD0, SIMD1, SIMD2 and SIMD3) (and specify their addresses). Each SIMD can write the output of the operation performed within the SIMD to a specific number of output registers within the output circuit 420.

[0136] Fig. 20 3 is a block diagram showing each SIMD engine (SIMD0, SIMD1, SIMD2 and SIMD3) connected to a corresponding memory bank 20000-20003, wherein each memory bank includes k rows, wherein each row forms an output register. In the case of an accumulation operation, the control logic 340 specifies a row address within each of the memory banks 20000-20003 so that the previously stored accumulated value is read from the addressed output register of the memory bank 20000-20003 and provided to the corresponding SIMD engine SIMD0-SIMD3. In response, the SIMD engine (SIMD0-SIMD3) performs a multiplication-accumulation operation to generate an updated accumulated value, which is then written back to the addressed output register within the corresponding memory bank 20000-20003.

[0137] As described in more detail below, the row address of the output register associated with each SIMD can be considered an input signal to the SIMD engine.The row address is the index of the row within the SIMD (referred to as the relative index within the SIMD).

[0138] Fig.21 2 is a block diagram showing register selection logic 2101 for determining the manner in which the contents of output register groups 20000, 20001, 20002, and 20003 are provided to SIMD engines SIMD0, SIMD1, SIMD2, and SIMD3, respectively. The register selection logic 2101 is implemented within the state machine and scheduler 431 of the control logic 430. In one embodiment, the register selection logic 2101 includes four register selection entries R0, R1, R2, and R3 that store row address values ​​that specify which of the output registers in the output register groups 20000, 20001, 20002, and 20003 have their contents provided to SIMD0, SIMD1, SIMD2, and SIMD3, respectively (or store values ​​received from SIMD0, SIMD1, SIMD2, and SIMD3, respectively). In the illustrated example, four register selection entries R0, R1, R2, and R3 specify output registers in row 1, row (K-1), row 0, and row 2 of output register banks 20000, 20001, 20002, and 20003, respectively (indicating that the contents of these output registers are provided to SIMD0, SIMD1, SIMD2, and SIMD3, respectively). Note that the output registers selected are in Fig.21 In subsequent cycles, the register select can be changed by changing the register select entries R0, R1, R2, and R3.

[0139] Unified Architecture

[0140] Fig. 2222 is a block diagram of a hardware system 2200 that unifies the various architectures and features presented above for different operands and outputs. The system 2200 includes (1) an operand block 2210 that includes one or more register files for storing each of the operands (operand A and operand B), (2) an input distribution block (IDB) 2220 that includes one or more buffers for each of the operands and a logic block for each of the operands, (3) a SIMD block 2230 that includes one or more SIMD engines, and (4) an output block 2240 that includes one or more output register files for each of the SIMD engines. In the illustrated system 2200, the operand block 2210 includes (one or more) operand A register files 2211 and (one or more) operand B register files 2212, which can be used to implement the various embodiments of the operand A register files 411 and operand B register files 412 described above. The input distribution block (IDB) 2220 includes an operand A IDB buffer 2221 and an operand A IDB logic 2223, which can be used to implement various embodiments of the above-mentioned operand A distribution circuit 416. The input distribution block 2220 also includes an operand B IDB buffer 2222 and an operand B IDB shift logic 2224, which can be used to implement various embodiments of the above-mentioned operand B distribution circuit 417. The SIMD block 2240 (which can be used to implement various embodiments of the above-mentioned SIMD block 401) includes SIMD0, SIMD1, SIMD2 and SIMD3. The output block 2240 (which can be used to implement various embodiments of the above-mentioned output circuit 420) includes output register files 2241, 2242, 2243 and 2244 respectively connected to SIMD0, SIMD1, SIMD2 and SIMD3.

[0141] Operand block 2210 is responsible for holding operand data. Based on the control signals explained below, data from operand register files 2211 and 2212 are passed to input allocation block 2220. In one embodiment of the unified architecture of system 2200, operand A register file 2211 includes one register file, and operand B register file 2212 includes four register files. In addition, the four register files for operand B allow two simultaneous reads (i.e., the memory is divided into two bodies). Although a general system can contain any number of register files for each of the operands, the optimal hardware should be designed so that the minimum number of register files is used but all required operations are supported.

[0142] In the illustrated illustration, three control signals OP_A_RF_SRC_ADDR_SEL, OP_A_RF_DEST_ADDR_SEL, and OP_A_RF_LOAD_FLAG are used to control the operation of operand A register file 2211. Similarly, three control signals OP_B_RF_SRC_ADDR_SEL, OP_B_RF_DEST_ADDR_SEL, and OP_B_RF_LOAD_FLAG are used to control the operation of operand B register file 2212.

[0143] The OP_A_RF_LOAD_FLAG and OP_B_RF_LOAD_FLAG signals specify whether data needs to be transferred from the operand A register file 2211 and the operand B register file 2212 to the operand A IDB buffer 2221 and the operand B IDB buffer 2222, respectively. If the OP_A_RF_LOAD_FLAG signal has a value of 1, two associated control signals (OP_A_RF_SRC_ADDR_SEL and OP_A_RF_DEST_ADDR_SEL) specify the source address and destination address for the operand A data. Similarly, if the OP_B_RF_LOAD_FLAG signal has a value of 1, two associated control signals (OP_B_RF_SRC_ADDR_SEL and OP_B_RF_DEST_ADDR_SEL) specify the source address and destination address for the operand B data. Note that not all embodiments will require destination addresses for operand A data and operand B data (i.e., if there is only one possible destination for operand A data or operand B data). If the OP_A_RF_LOAD_FLAG signal or the OP_B_RF_LOAD_FLAG signal has a value of 0, data is not read or transferred from the corresponding operand A register file 2211 or operand B register file 2212. The OP_A_RF_LOAD_FLAG signal and the OP_B_RF_LOAD_FLAG signal may be generated by the state machine and scheduler 431 of the control logic 430.

[0144] The OP_A_RF_SRC_ADDR_SEL signal specifies the row address in the (one or more) operand A register file 2211 to be read. In the above-mentioned operation mode, the OP_A_RF_SRC_ADDR_SEL signal will only include one address, which specifies the operand A register file to be read. The OP_B_RF_SRC_ADDR_SEL signal specifies the (one or more) row addresses in the (one or more) operand B register file 2212 to be read. Depending on the operation mode, the OP_B_RF_SRC_ADDR_SEL signal can be only one address (architecture 1B or 2B) or four addresses (architecture 3B) or 8 addresses (architecture 4B). The hardware has appropriate modes to handle different situations, where these modes are specified by the control register 432 of the control logic 430. The above-mentioned source address can be generated in hardware by the state machine and scheduler 431 of the control logic 430.

[0145] The OP_A_RF_DEST_ADDR_SEL signal specifies the destination address in the operand AIDB buffer 2221 to which the data read from the operand A register file 2211 is passed. Similarly, the OP_B_RF_DEST_ADDR_SEL signal specifies the destination address in the operand B IDB buffer 2222 (or shift logic 2224) to which the data read from the operand B register file 2212 is passed. Again depending on the mode of operation, these addresses can be a single address or multiple addresses. The addresses can be generated in hardware by the state machine and scheduler 431 of the control logic 430.

[0146] Note that multiple SIMD engines share operand register files and control logic. This results in higher compute density, i.e., more compute power per unit silicon area. The sharing of operand register files and control logic also saves power and SRAM bandwidth. The SRAM bandwidth savings come from the fact that only two operand register files need to be written to support multiple SIMD engines.

[0147] The input distribution block 2220 includes buffers 2221-2222 for holding operand data and logic blocks 2223-2224 for manipulating the data. Fig. 22As illustrated, operand A values ​​from register file 2211 are stored in buffer 2221 before being provided to operand AIDB logic 2223 (which performs the switching / demultiplexing function described above). In contrast, operand B values ​​from register file 2212 are routed through shift logic 2224 before being stored in buffer 2222. Thus, for operand A, the data is first loaded and then manipulated to obtain input to SIMD engine 2230, while for operand B, the data is first manipulated (shifted) and then buffered.

[0148] Instead of feeding the SIMD engines directly from the operand register file, the input distribution block 2220 acts as a small cache from which the operands are fed to the SIMD engines. The input distribution block 2220 allows multiple SIMD engines to be run in parallel using a single control circuit. As described above, data from the operand A register file 2211 can be manipulated so that it provides different data to multiple SIMD engines.

[0149] The OP_A_IDB_ADDR_SEL signal specifies the address of the operand AIDB buffer to be used for each SIMD. In all the architectures discussed, we only used one register file word data for operand A, but in general, buffer 2221 can hold multiple register file words, and each of the SIMD engines may select a different register file word from buffer 2221. Typically, there is only one operand A buffer in the optimal architecture. However, this single buffer is actually implemented using a double buffer so that the same buffer is not read and written in one cycle. The hardware manages the double buffer in a transparent manner. Therefore, in most cases, this signal is managed internally by the hardware.

[0150] The OP_A_IDB_DATA_SEL signal of the control operand AIDB logic 2223 specifies the data that needs to be passed to each SIMD. For example, in architecture 2A, a single value is effectively copied and spread to all SIMD engines. This signal specifies the index of the value that needs to be copied. Similarly, in architecture 3A, four consecutive values ​​are obtained from the register file word, and each of these values ​​is effectively copied and sent to a SIMD. In this case, the OP_A_IDB_DATA_SEL signal specifies the index of the value that needs to be copied for SIMD0. For other SIMD engines (SIMD1-SIMD3), the index value is incremented. For architecture 1A, since the complete register word stored in buffer 2221 is sent to all SIMD engines, the OP_A_IDB_DATA_SEL signal is not required.

[0151] The OP_B_IDB_SHIFT_SEL signal that controls operand B IDB shift logic 2224 is used to control the manner in which register file words received from operand B register file 2212 are shifted (i.e., when two register file words from the same register file are read for operand B). Note that the OP_B_IDB_SHIFT_SEL signal (and operand B IDB shift logic 2224) is only required if the system 2200 implements architecture 4B. In this case, the OP_B_IDB_SHIFT_SEL signal specifies how two register file words need to be manipulated to produce one register file word (in the manner described above).

[0152] Convolution operations typically involve data shifting. Positioning operand B shift logic 2224 between operand B register file 2212 and the SIMD engine advantageously reduces hardware overhead by allowing data to be read from operand B register file 2212 multiple times, each time applying a different shift to the data. If shift logic 2224 is not implemented in this manner, the shifted data will need to be written to operand B register file 2212 and therefore cannot be reused multiple times as in the proposed architecture.

[0153] The OP_B_IDB_ADDR_SEL value specifies the address of the operand B IDB buffer that will provide its contents as input for each of the SIMD engines. Fig.16 This is illustrated for Architecture 2B in . This is one of the most important signals in the architecture and provides a lot of flexibility in the types of computations that can be done. The OP_B_IDB_ADDR_SEL value typically comes from the state machine and scheduler 431 for the mode corresponding to Architecture 2B, but can also be managed by hardware if the data access pattern is predictable.

[0154] The use of multiple operand B buffers 2222 in the input distribution block 2220 allows different SIMD engines to obtain different operand B data in a given cycle. The use of four operand B buffers 2222 (i.e., the same as the number of SIMD engines) allows four simultaneous reads so that each SIMD receives different data. This is much cheaper (from a hardware perspective) than implementing the operand B register file 2212 (which will also allow four simultaneous read operations to supply SIMD0-SIMD3) with a four-port memory. Providing an operand B buffer 2222 to buffer a small amount of register words from the operand B register file 2212 effectively provides a small high-speed cache that can be accessed by any one of SIMD0-SIMD3. This provides a good compromise between hardware complexity and the flexibility required for certain classes of algorithms.

[0155] SIMD block 2230 includes one or more SIMD engines that perform actual calculations on the data provided by input distribution block 2220. Because SIMD engines can support different types of operations, the operations to be performed should be provided as input. Therefore, the SIMD_OPERATION_SEL value is used to specify the operations to be performed by the SIMD engine. In theory, different SIMD engines can perform different operations, but in general, the same operation selection value SIMD_OPERATION_SEL is used to drive all SIMD engines.

[0156] The results of the calculations performed by the SIMD engine need to be written to the output register files 2241-2244 within the output block 2240. In addition, for operations such as accumulation, the previously accumulated value needs to be read from the output register files 2241-2244 (and provided to the SIMD engine). Typically, the accumulated value is written back to the same location as the previously accumulated value. However, for generality, two control values ​​OUTPUT_RF_ADDR_SEL_0 and OUTPUT_RF_ADDR_SEL_1 are provided to the output block 2240, allowing the read address and write address of each of the output register files 2241-2244 to be specified separately. In one embodiment, the control value OUTPUT_RF_ADDR_SEL_0 specifies the write address of each of the output register files 2241-2244, and the control value OUTPUT_RF_ADDR_SEL_1 specifies the read address of each of the output register files 2241-2244. When discussing the architecture for the output circuit 420, the use Fig.21 Gives a graphical illustration of the specified output address.

[0157] Note that including multiple output registers in each of the output register files 2241-2244 advantageously provides flexibility with respect to the types of operations that can be performed by the system architecture. Some examples of this flexibility are described in more detail below.

[0158] Various examples of computer architectures for operating certain embodiments according to the present invention will now be described.

[0159] Fig.23 is a block diagram of a computer architecture 2300 that may be used to perform matrix multiplication according to one embodiment of the present invention.

[0160] Fig.24 Is a diagram to be Fig.23FIG. 23 is a diagram of a computer architecture 2300 for multiplying two matrices I and J. Matrix I has 64 rows and 16 columns, and matrix J has 16 rows and 4 columns. Each row of matrix I can represent a weight vector, and each column of matrix J can represent an activation vector in a machine learning system. As illustrated, matrix I includes 1024 (32-bit) values ​​w 0,0 to w 63,15 As shown in the figure, the matrix J includes 64 (32-bit) values ​​a0-a 15 、b0-b 15 、c0-c 15 and d0-d 15 .

[0161] Matrix I and matrix J are stored in system memory 440 ( Figure 4 )middle. Fig.25 4 is a block diagram illustrating the manner in which the contents of matrix I and matrix J are logically stored within system memory 440. Matrix I is stored in operand A memory block 441, which includes 256 rows, each row including four weight values. For example, the first row (row 0) of operand A memory block 441 includes weight values ​​[w 0,0 、w 1,0 、w 2,0 、w 3,0 ]. The first column (column 0) of matrix I (i.e., the value w i,0 , where i=0 to 63) are stored in the first sixteen consecutive rows of operand A memory block 441. The second column (column 1) of matrix I (i.e., the value w i,1 , where i=0 to 63) are stored in a second set of sixteen consecutive rows of operand A memory block 441. The remaining columns of matrix I (column 2 - column 15) are stored in consecutive sets of sixteen consecutive rows within operand A memory block 441, as illustrated.

[0162] Matrix J is stored in operand B memory block 442 including 16 rows, each row including four activation values. For example, the first row of operand B memory block 442 includes the activation values ​​[d0, c0, b0, a0] included in the first row of matrix J. The remaining rows (rows 1-15) of matrix J are stored in consecutive rows (rows 1-15) of operand B memory block 442.

[0163] The multiplication of matrix I and matrix J is performed as follows.

[0164] State Machine and Scheduler 431 ( Figure 4 ) causes operand packing logic 433 to retrieve entry w from the first row of operand A memory block 441 0,0 、w 1,0 、w 2,0 and w 3,0, and retrieves entries a0, b0, c0, and d0 from the first row of operand B memory block 442. The state machine and scheduler 431 stores the retrieved entries w 0,0 、w 1,0 、w 2,0 and w 3,0 The operand A register file 411 is written, and the retrieved entries a0, b0, c0, and d0 are written to the operand B register file 412. The result is Fig.23 The operand A distribution circuit 416 within the input distribution block 415 is controlled to distribute the entry w 0,0 、w 1,0 、w 2,0 and w 3,0 As 'operand A', it is routed from operand A register file 411 to SIMD0, SIMD1, SIMD2 and SIMD3 respectively (as shown above Fig.11 The operand B distribution circuit 417 within the input distribution block 415 is controlled to route entries a0, b0, c0, and d0 from the operand B register file 412 to each of SIMD0, SIMD1, SIMD2, and SIMD3 (in the manner specified above). Fig.14 specified method).

[0165] Each of the SIMD engines (SIMD0-SIMD3) multiplies corresponding entries of operand A and operand B (e.g., SIMD0 performs (a0 x w 0,0 )、(b0 xw 0,0 )、(c0 xw 0,0 ) and (d0 xw 0,0 )) to generate the corresponding product.

[0166] Fig.26 The diagram illustrates the mapping of the contents of the output register groups 20000, 20001, 20002, and 20003 according to this example. Each entry of the output register groups 20000, 20001, 20002, and 20003 is initially set to a zero value. Each entry of the output register groups 20000, 20001, 20002, and 20003 corresponds to a dot product of a matrix multiplication. Each dot product is specified by a row of the matrix I and a column of the matrix J. For example, the matrix labeled (w 0,i a i )'s output register bank 20000 stores row 0 (w 0,0 、w 0,1 、w 0,2 ... 0,15 ) and the columns 0 (a0, a1, a2, ... a 15 ). Similarly, 26,id i )'s output register set 20002 stores row 26 (w 26,0 、w 26,1 、w 26,2 ... 26,15 ) and the columns 3 (d0, d1, d2, ... d 15 )’s dot product J.

[0167] The state machine and scheduler 431 controls the addressing of the output register groups 20000, 20001, 20002 and 20003 in parallel. During the initial calculation (as described above and in Fig.23 ), the state machine and scheduler 431 addresses row 0 of each of the output register banks 20000-20003. As a result, the zero values ​​stored in row 0 of the output register banks 20000-20003 are provided to SIMD0-SIMD3, respectively.

[0168] During the initial calculation, each of SIMD0, SIMD1, SIMD2 and SIMD3 performs an accumulation operation in which a zero value retrieved from output register bank 20000-20003 is added to the product calculated by SIMD0-SIMD3. The accumulated value is then written back to row 0 of the corresponding output register bank 20000-20003.

[0169] For example, the entry (w 0,i d i )、(w 0,i c i )、(w 0,i b i ) and (w 0,i a i ) is provided to SIMD0. SIMD0 then calculates the product (w 0,0 x d0)、(w 0,0 x c0)、(w 0,0 x b0) and (w 0, 0x a0) are added to these retrieved zero values ​​to create the accumulated values. SIMD0 then writes these accumulated values ​​back to the entries of row 0 of output register bank 20000 (w 0,i d i )、(w 0,i ci)、(w 0,i b i ) and (w 0,i a i ). Similar operations are performed by SIMD1-SIMD3.

[0170] The state machine and scheduler 431 then increments the address used to access the operand A memory block 441, resulting in the value of the next row (i.e., w 4,0 、w 5,0 、w 6,0 and w 7,0 ) is retrieved and stored in the operand A register file 411. The operand A allocation circuit 416 is combined with the above Fig.23 These received values ​​are routed in the same manner as described above. That is, the value w 4,0 、w 5,0 、w 6,0 and w 7,0 They are provided to SIMD0, SIMD1, SIMD2 and SIMD3 respectively. At this time, operand B remains unchanged.

[0171] Each of the SIMD engines (SIMD0-SIMD3) multiplies corresponding entries of operand A and operand B (e.g., SIMD0 performs (a0 x w 4,0 )、(b0 xw 4,0 )、(c0 xw 4,0 ) and (d0 xw 4,0 ))thereby providing the corresponding product.

[0172] During this second computation, the state machine and scheduler 431 increments the row address of each of the output register banks 20000-20003, thereby addressing row 1 within each of these output register banks. As a result, the zero values ​​stored in row 1 of the output register banks 20000-20003 are provided to SIMD0-SIMD3.

[0173] During the second calculation, each of SIMD0, SIMD1, SIMD2 and SIMD3 performs an accumulation operation in which the zero value retrieved from the output register bank 20000-20003 is added to the product calculated by the SIMD engine. The accumulated value is then written back to the output register bank 20000-20003.

[0174] For example, the entry (w 4,i d i )、(w 4,i c i )、(w 4,i b i ) and (w 4,i a i ) is provided to SIMD0. SIMD0 then calculates the product (w 4,0 x d0)、(w 4,0 x c0)、(w 4,0 x b0) and (w4, 0x a0) are added to these retrieved zero values ​​to create the accumulated values. SIMD0 then writes these accumulated values ​​back to the entries of row 1 of output register bank 20000 (w 4,i d i )、(w 4,i c i )、(w 4,i b i ) and (w 4,i a i ). Similar operations are performed by SIMD1-SIMD3.

[0175] The above process is repeated until the operand A distribution circuit 416 allocates all (64) weight values ​​w in the above manner. 0,0 to w 63,0 The first column (column 0) of matrix I is routed to SIMD0-SIMD3 in sequence as operand A values.

[0176] After the weight values ​​from the first column (column 0) of matrix I have been used to perform the multiply-accumulate operation (e.g., before being combined with the value w 0,0 to w 63,0 After the associated product has been calculated), the state machine and scheduler 431 resets the address of the output register bank 20000-20003 to row 0. In addition, the state machine and scheduler 431 increments the address used to access the operand A memory block 441 so that the value (w 0,1 、w 1,1 、w 2,1 、w 3,1 ) are retrieved and stored in operand A register file 411. Operand A allocation circuit 416 allocates these values ​​(w 0,1 、w 1,1 、w 2,1 、w 3,1 ) are routed to SIMD0, SIMD1, SIMD2 and SIMD3 respectively (to combine with the above Fig.23 described in the same manner).

[0177] The state machine and scheduler 431 also increments the address used to access the operand B memory block 442 by 1, so that the values ​​(a1, b1, c1, d1) from row 1 of the operand B memory block 442 are retrieved and stored in the operand B register file 412. The operand B distribution circuit 417 routes these values ​​(a1, b1, c1, d1) to each of SIMD0, SIMD1, SIMD2, and SIMD3 (in the same manner as in Fig.23 The values ​​in (a0, b0, c0, d0) were previously routed to SIMD0, SIMD1, SIMD2, and SIMD3 in the same manner).

[0178] SIMD0, SIMD1, SIMD2, and SIMD3 perform multiply-accumulate operations on the received values, and the results are stored in row 0 of output registers 20000, 20001, 20002, and 20003, respectively, in the manner described above.

[0179] The state machine and scheduler 431 then increments the address used to access the operand A memory block 441, resulting in the value of the next row (i.e., w 4,0 、w 5,0 、w 6,0 and w 7,0 ) is retrieved and stored in the operand A register file 411. The operand A allocation circuit 416 is combined with the above Fig.23 These received values ​​are routed in the same manner as described above. That is, the value w 4,0 、w 5,0 、w 6,0 and w 7,0 are provided to SIMD0, SIMD1, SIMD2 and SIMD3 respectively. At this time, operand B (a1, b1, c1, d1) remains unchanged. SIMD0, SIMD1, SIMD2 and SIMD3 perform multiplication-accumulation operations on these received values, and the results are stored in row 1 of output registers 20000, 20001, 20002 and 20003 respectively in the above manner.

[0180] The above process is repeated until the operand A distribution circuit 416 allocates all (64) weight values ​​w 0,1 to w 63,1 The second column (column 1) of matrix I is routed to SIMD0-SIMD3 in sequence as operand A values ​​(while operand B (a1, b1, c1, d1) remains unchanged).

[0181] The above process is then repeated so that a multiplication-accumulation operation is performed on each column (column 0 to column 15) of matrix I and each row (row 0 to row 15) of matrix J. At the end of this process, output register bank 20000-20003 stores the dot product of each row of matrix I and each column of matrix J. For example, the entry (w 0,i a i ) stores the dot product of row 0 of matrix I and column 0 of matrix J, and outputs the entry (w 29,i d i ) stores the dot product of row 29 of matrix I and column 3 of matrix J.

[0182] Advantageously, the present invention provides an efficient structure for multiplying matrix I and matrix J. SIMD0-SIMD3 provides a high degree of processing parallelism (e.g., 16 parallel multiplication-accumulation operations at a time), which advantageously reduces the time required for performing matrix multiplication. In addition, the control circuit system required for implementing matrix multiplication is advantageously simple. The address inputs to operand A memory block 441 and output register group 20000-20003 are simply incremented after each multiplication-accumulation operation, and the address inputs to operand B memory block 442 are simply incremented after every 16 multiplication-accumulation operations. The input distribution block 415 advantageously maintains the same configuration during the entire matrix multiplication.

[0183] A matrix that contains a large number of zero-valued entries is called a "sparse" matrix. For example, a matrix that includes 7 / 8 or more zero-valued entries can be called a sparse matrix. Multiplication involving sparse matrices can involve a large number of unnecessary operations. In the example provided above, assume that 7 / 8 of the entries of matrix I include zero values. In this case, only 512 (16 x 16 x 16 x (1 / 8)) multiply-accumulate operations are required to multiply matrix I and matrix J. However, all 4096 operations (16 x 16 x 16) above will be combined with the above Figure 23-Figure 26 Therefore, according to another embodiment of the present invention, there is provided a method for using Figure 23-Figure 26 A structure with methods for performing multiplication with sparse matrices.

[0184] Assume that matrix I is a sparse matrix, in which only one eighth of the entries of matrix I have non-zero values. As described above, the processing is performed sequentially for each column of matrix I (e.g., initially processing column 0 of matrix I, then column 1 of matrix I, etc.). Therefore, the processing of the first column of matrix I will be described, and it should be understood that the remaining columns of matrix I are processed in the same manner.

[0185] In a first example, assume that only the following eight entries (out of a total of 64 entries) of column 0 of matrix I have non-zero values: 3,0 、w 5,0 、w 8,0 、w 10,0 、w 11,0 、w 24,0 、w 58,0 and w 61,0 .

[0186] Initially, operand packing logic 433 identifies the row address of the non-zero value within matrix I. Thus, in this example, operand packing logic 433 determines that the non-zero value w 3,0 、w 5,0 、w 8,0 、w 10,0 、w 11,0 、w24,0 、w 58,0 and w 61,0 The row addresses of the operands 20000-20003 are located in rows 3, 5, 8, 10, 11, 24, 58, and 61 of the matrix I, respectively. Using the row address information, the operand packing logic 433 determines which of the output register banks 20000-20003 is used to store the dot product associated with the identified non-zero value. Typically, this determination is made by dividing the row address of the non-zero value within the operand matrix I by "4" and then using the remainder (R) of this division operation to identify the output register bank (where the remainder (R) identifies the output register bank 20000-20003). R ).

[0187] Operand packing logic 433 also determines the row within the output register bank in which the dot product is stored. Typically, this determination is made by dividing the row address of the non-zero value within matrix I by "4" and ignoring the remainder (R).

[0188] In this example, a non-zero value w 8,0 and w 24,0 are located in rows 8 and 24 of matrix I. Dividing these row numbers by 4 gives a remainder of "0", indicating a non-zero value w 8,0 and w 24,0 The dot product of is located in output register bank 20000. Furthermore, since 8 / 4=2 and 24 / 4=6, the non-zero value w 8,0 and w 24,0 The dot products of are located in rows 2 and 6 of output register bank 20000. This result is given by Fig.26 Confirming that it is shown that the output register set 20000 includes the dot product [(w 8,i d i )、(w 8,i c i )、(w 8,i b i )、(w 8,i a i )], and the dot product [(w 24,i d i )、(w 24,i c i )、(w 24,i b i )、(w 24,i a i )].

[0189] In this example, a non-zero value w 5,0 and w 61,0 Located in rows 5 and 61 of matrix I. Dividing these row numbers by 4 gives a remainder of "1", indicating a non-zero value w5,0 and w 61,0 The dot product of is located in output register bank 20001. Furthermore, since 5 / 4 = 1 (remainder 1) and 61 / 4 = 15 (remainder 1), the non-zero value w 5,0 and w 61,0 The dot product of is located in rows 1 and 15 of output register bank 20001. This result is given by Fig.26 Confirming, it is shown that the output register set 20001 includes the dot product [(w 5,i d i )、(w 5,i c i )、(w 5,i b i )、(w 5,i a i )], and the dot product [(w 61,i d i )、(w 61,i c i )、(w 61,i b i )、(w 61,i a i )].

[0190] In this example, a non-zero value w 10,0 and w 58,0 Located in rows 10 and 58 of matrix I. Dividing these row numbers by 4 gives a remainder of "2", indicating a non-zero value w 10,0 and w 58,0 The dot product of is located in output register bank 20002. Furthermore, since 10 / 4=2 (remainder 2) and 58 / 4=14 (remainder 2), the non-zero value w 10,0 and w 58,0 The dot products of are located in rows 2 and 14 of output register bank 20002. This result is given by Fig.26 Confirming, it is shown that the output register set 20002 includes the dot product [(w 10,i d i )、(w 10,i c i )、(w 10,i b i )、(w 10,i a i )], and the dot product [(w 58,i d i )、(w 58,i c i )、(w 58,i bi )、(w 58,i a i )].

[0191] In this example, a non-zero value w 3,0 and w 11,0 are located in rows 3 and 11 of matrix I. Dividing these row numbers by 4 gives a remainder of "3", indicating a non-zero value w 3,0 and w 11,0 The dot product of is located in output register bank 20003. Furthermore, since 3 / 4 = 0 (remainder 3) and 11 / 4 = 2 (remainder 3), the non-zero value w 3,0 and w 11,0 The dot product of is located in row 0 and row 2 of output register bank 20003. The result is given by Fig.26 Confirming, it is shown that the output register set 20003 includes the dot product [(w 3,i d i )、(w 3,i c i )、(w 3,i b i )、(w 3,i a i )], and the dot product [(w 11,i d i )、(w 11,i c i )、(w 11,i b i )、(w 11,i a i )].

[0192] After making the above determination, the operand packing logic 433 converts the non-zero value w of column 0 of matrix I into 3,0 、w 5,0 、w 8,0 、w 10,0 、w 11,0 、w 24,0 、w 58,0 and w 61,0 Sort (pack) into operand A memory block 441. See new statement later in document.

[0193] The first non-zero value that causes the dot product to be stored in output register banks 20000, 20001, 20002, and 20003 is stored in the first row (row 0) of operand A memory block 441. Thus, in this example, the non-zero values ​​w that cause the dot product to be in output register banks 20000, 20001, 20002, and 20003, respectively, are stored in the first row (row 0) of operand A memory block 441. 8,0 、w 5,0 、w10,0 and w 3,0 Stored in row 0 of operand A memory block 441.

[0194] The next non-zero value that causes the dot product to be stored in the output register bank 20000, 20001, 20002, and 20003 is stored in the second row (row 1) of the operand A memory block 441. Thus, in this example, the non-zero values ​​w that cause the dot product to be in the output register bank 20000, 20001, 20002, and 20003, respectively, are stored in the second row (row 1) of the operand A memory block 441. 24,0 、w 61,0 、w 58,0 and w 11,0 The above sorting / packing of the non-zero values ​​of column 0 of matrix I into operand A memory block 441 is performed in Fig. 27 The diagram in the figure shows the following figure.

[0195] Initially, the state machine and scheduler 431 causes the first row of operand A memory block 441 and operand B memory block 442 to be retrieved and loaded into operand A register file 411 and operand B register file 412, respectively, as indicated by Fig. 27 Therefore, the operand A register file 411 stores the non-zero weight values ​​w of the matrix I 8,0 、w 5,0 、w 10,0 and w 3,0 , and operand B register file 412 stores activation values ​​d0, c0, b0, and a0 of matrix J. State machine and scheduler 431 causes operand A allocation circuit 416 to allocate non-zero value w 8,0 、w 5,0 、w 10,0 and w 3,0 The operand A register file 411 is routed to SIMD0, SIMD1, SIMD2, and SIMD3 as operand A. At the same time, the state machine and scheduler 431 causes the operand B distribution circuit 417 to route the values ​​d0, c0, b0, and a0 to each of the SIMD engines as operand B. These values ​​d0, c0, b0, and a0 are routed from row 0 of the operand B memory block 442 (i.e., row 0 of the matrix J) because the operand A value w 8,0 、w 5,0 、w 10,0 and w 3,0 Each of comes from column 0 of matrix I. Fig. 27 is a block diagram illustrating the above-described configuration.

[0196] SIMD0, SIMD1, SIMD2 and SIMD3 multiply the received operand A and operand B in the above manner. The state machine and scheduler 431 responds to the non-zero value w 8,0、w 5,0 、w 10,0 and w 3,0 The previously determined rows in the associated output register banks 20000, 20001, 20002, and 20003 are addressed independently. That is, the state machine and scheduler 431 addresses row 2, row 1, row 2, and row 0 within the output register banks 20000, 20001, 20002, and 20003, respectively. As described above, all rows of the output register banks initially store a "0" value.

[0197] SIMD0, SIMD1, SIMD2, and SIMD3 perform an accumulation operation in which the calculated products are added to the zero values ​​retrieved from the output register banks 20000, 20001, 20002, and 20003, respectively. SIMD0, SIMD1, SIMD2, and SIMD3 then write the accumulated values ​​to the addressed rows (row 2, row 1, row 2, and row 0, respectively) of the output register banks 20000, 20001, 20002, and 20003, respectively.

[0198] like Fig.28 As illustrated, the state machine and scheduler 431 then retrieves the non-zero value w from the second row of the operand A memory block 441. 24,0 、w 61,0 、w 58,0 and w 11,0 , and store these non-zero values ​​in operand A register file 411.

[0199] Therefore, operand A register file 411 stores the non-zero weight values ​​w of matrix I 24,0 、w 61,0 、w 58,0 and w 11,0 , and operand B register file 412 stores activation values ​​d0, c0, b0, and a0 of matrix J. State machine and scheduler 431 causes operand A allocation circuit 416 to allocate non-zero value w 24,0 、w 61,0 、w 58,0 and w 11,0 The operand A register file 411 is routed to SIMD0, SIMD1, SIMD2, and SIMD3 as operand A, respectively. At the same time, the state machine and scheduler 431 continues to cause the operand B distribution circuit 417 to route the values ​​d0, c0, b0, and a0 to each of the SIMD engines as operand B. These values ​​d0, c0, b0, and a0 are routed from row 0 of the operand B memory block 442 (i.e., row 0 of the matrix J) because the operand A value w 24,0 、w 61,0 、w 58,0 and w 11,0 Each of comes from column 0 of matrix I. Fig.28 is a block diagram illustrating the above-described configuration.

[0200] SIMD0, SIMD1, SIMD2 and SIMD3 multiply the received operand A and operand B in the above manner. The state machine and scheduler 431 responds to the non-zero value w 24,0 、w 61,0 、w 58,0 and w 11,0 The previously determined rows in the associated output register banks 20000, 20001, 20002 and 20003 are addressed independently. That is, the state machine and scheduler 431 addresses row 6, row 15, row 14 and row 2 within the output register banks 20000, 20001, 20002 and 20003, respectively.

[0201] SIMD0, SIMD1, SIMD2, and SIMD3 perform an accumulation operation in which the calculated products are added to the zero values ​​retrieved from the output register banks 20000, 20001, 20002, and 20003, respectively. SIMD0, SIMD1, SIMD2, and SIMD3 then write the accumulated values ​​to the addressed rows (row 6, row 15, row 14, and row 2, respectively) of the output register banks 20000, 20001, 20002, and 20003, respectively.

[0202] The above operations complete the processing of the first column (column 0) of matrix I. The same processing steps are performed on each of the remaining 15 columns of matrix I, thereby completing the multiplication of the "sparse" matrix I and matrix J. Note that in these processing steps, the non-zero values ​​in column 1 of matrix I are multiplied by the values ​​in row 1 of matrix J (i.e., a1, b1, c1, and d1), the non-zero values ​​in column 2 of matrix I are multiplied by the values ​​in row 2 of matrix J (i.e., a2, b2, c2, and d2), and so on). Advantageously, the SIMD engine remains busy (i.e., performing multiplication-accumulation operations on non-zero matrix values) while minimizing the number of multiplication-accumulation operations required to perform the multiplication of the "sparse" matrix I and matrix J. In the foregoing manner, the computer architecture performs multiplication of sparse matrices in an efficient (and fast) manner.

[0203] The above sparse matrix multiplication example includes non-zero values ​​of matrix I uniformly distributed between output register groups 20000-20003 (e.g., eight non-zero values ​​are distributed in column 0 of matrix I, so that each of output register groups 20000-20003 is associated with two non-zero values). However, in other examples, the distribution of non-zero values ​​may not be so uniform. Another embodiment of the present invention handles the non-uniform distribution of non-zero values ​​in a manner described in more detail below.

[0204] Assume that the first sixteen non-zero entries in the first three columns of matrix I are entries w 2,0 、w12,0 、w 32,0 、w 38,0 、w 45,0 、w 56,0 (in column 0 of matrix I), w 7,1 、w 14,1 、w 21,1 、w 25,1 、w 37,1 、w 43,1 (in column 1 of matrix I), w 8,2 、w 10,2 、w 23,2 and w 51，2 (in column 2 of matrix I).

[0205] Operand packing logic 433 identifies the row addresses of non-zero values ​​within matrix I (e.g., non-zero entries w 2,0 2 of matrix I). Using this row address information, operand packing logic 433 determines which of output register banks 20000-20003 to use to store the dot product associated with the identified non-zero value in the manner described above. Operand packing logic 433 also determines the row within the output register bank in which to store the dot product in the manner described above.

[0206] Therefore, in this example, operand packing logic 433 determines the value of the operand packing logic 433 associated with the non-zero entry w 12,0 、w 32,0 、w 56,0 and w 8,2 The associated dot products are mapped to rows 3, 8, 14, and 2 of output register bank 20000, respectively; together with the non-zero entries w 45,0 、w 21,0 、w 25,0 and w 37,2 The associated dot products are mapped to rows 11, 5, 6, and 9 of output register bank 20001, respectively; together with the non-zero entries w 2,0 、w 38,0 、w 14,1 and w 10,2 The associated dot products are mapped to rows 0, 9, 3, and 2 of output register bank 20002, respectively; and are combined with the non-zero entries w 7,1 、w 43,1 、w 23,2 and w 51,2 The associated dot products are mapped to rows 7, 10, 5, and 12 of output register bank 20003, respectively.

[0207] Note that the three non-zero entries in column 0 of matrix I (w 12,0 、w 32,0 and w 56,0 ) is mapped to the output register bank 20000, a non-zero entry (w45,0 ) is mapped to the output register bank 20001, and the two non-zero entries (w 2,0 and w 38,0 ) is mapped to output register group 20002, and the non-zero entries of column 0 of matrix I are not mapped to output register group 20003.

[0208] The non-zero entries of column 1 of matrix I are not mapped to output register bank 20000. The three non-zero entries of column 1 of matrix I (w 21,1 、w 25,1 、w 37,1 ) is mapped to the output register set 20001, a non-zero entry (w 14,1 ) is mapped to the output register bank 20002, and the two non-zero entries (w 7,1 and w 43,1 ) is mapped to output register group 20003.

[0209] A non-zero entry (w 8,2 ) is mapped to the output register set 20000, the non-zero entry of column 2 of matrix I is not mapped to the output register set 20001, and a non-zero entry (w 10,2 ) is mapped to the output register bank 20002, and the two non-zero entries (w 23,2 and w 51,2 ) is mapped to output register group 20003.

[0210] After making the above determination, operand packing logic 433 sorts (packs) the non-zero values ​​of columns 0, 1, and 2 of matrix I into operand A memory block 441 as follows. The first non-zero value that causes the dot product to be stored in output register bank 20000, 20001, 20002, and 20003 is stored in the first row (row 0) of operand A memory block 441. Thus, in this example, the non-zero values ​​w that cause the dot product to be in output register bank 20000, 20001, 20002, and 20003, respectively, are stored in the first row (row 0) of operand A memory block 441. 12,0 、w 45,0 、w 2,0 and w 7,1 Stored in row 0 of operand A memory block 441.

[0211] The next non-zero value that causes the dot product to be stored in the output register bank 20000, 20001, 20002, and 20003 is stored in the second row (row 1) of the operand A memory block 441. Thus, in this example, the non-zero values ​​w that cause the dot product to be in the output register bank 20000, 20001, 20002, and 20003, respectively, are stored in the second row (row 1) of the operand A memory block 441. 32,0 、w21,1 、w 38,0 and w 23,2 Stored in row 1 of operand A memory block 441.

[0212] The next non-zero value that causes the dot product to be stored in the output register bank 20000, 20001, 20002, and 20003 is stored in the third row (row 2) of the operand A memory block 441. Thus, in this example, the non-zero values ​​w that cause the dot product to be in the output register bank 20000, 20001, 20002, and 20003, respectively, are stored in the third row (row 2) of the operand A memory block 441. 8,2 、w 37,1 、w 10,2 and w 43,1 Stored in row 2 of operand A memory block 441.

[0213] The above sorting / packing of the non-zero values ​​of columns 0, 1, and 2 of matrix I into operand A memory block 441 is performed in Fig.29 The diagram in the figure shows the following figure.

[0214] Initially, state machine and scheduler 431 causes row 0 of operand A memory block 441 to be retrieved and loaded into operand A register file 411 and then passed to operand A allocation circuit 416. Thus, operand A register file 411 stores the non-zero weight values ​​w of matrix I. 12,0 、w 45,0 、w 2,0 and w 7,1 Note that in alternative embodiments, these non-zero weight values ​​w 12,0 、w 45,0 、w 2,0 and w 7,1 Stored in a buffer within operand A allocation circuit 416.

[0215] The state machine and scheduler 431 also causes rows 0 and 1 of operand B memory block 442 to be retrieved and loaded into operand B register file 412, and then passed to operand B buffers B0 and B1, respectively, within operand B allocation circuit 417. Thus, operand B register file 412 and operand B buffer B0 store activation values ​​d0, c0, b0, and a0 of matrix J, and operand B register file 412 and operand B buffer B1 store activation values ​​d1, c1, b1, and a1 of matrix J. This situation is Fig.29 Shown in.

[0216] Note that state machine and scheduler 431 retrieves activation values ​​from row 0 and row 1 of operand B memory block 432, as these two activation values ​​are needed to compute the required dot products associated with the retrieved weight values ​​(taken from the first two columns of matrix I) included in operand A. Note also that operand B register file 412 may be loaded serially or in parallel from operand B memory block 442, and buffers B0-B3 of operand B allocation circuitry 417 may be loaded serially (or in parallel) from operand B register file 412. Fig.16 ) or in parallel ( Figure 17-Figure 18 )load.

[0217] The state machine and scheduler 431 causes the operand A allocation circuit 416 to convert the non-zero value w 12,0 、w 45,0 、w 2,0 and w 7,1 The operand A register file 411 is routed to SIMD0, SIMD1, SIMD2, and SIMD3 as operand A. At the same time, the state machine and scheduler 431 causes the operand B allocation circuit 417 to route the values ​​d0, c0, b0, and a0 to each of SIMD0, SIMD1, and SIMD2 as operand B, and also causes the operand B allocation circuit 417 to route the values ​​d1, c1, b1, and a1 to SIMD3. Fig.29 In the illustrated embodiment, operand B select register 1601 stores an operand B select signal that enables routing of these operand B values.

[0218] SIMD0, SIMD1, SIMD2 and SIMD3 multiply the received operand A and operand B in the above manner. The state machine and scheduler 431 responds to the non-zero value w 12,0 、w 45,0 、w 2,0 and w 7,1 The previously determined rows in the associated output register banks 20000, 20001, 20002, and 20003 are addressed independently. That is, the state machine and scheduler 431 addresses row 4, row 11, row 0, and row 1 within the output register banks 20000, 20001, 20002, and 20003, respectively. As described above, all rows of the output register banks initially store a "0" value.

[0219] SIMD0, SIMD1, SIMD2, and SIMD3 perform an accumulation operation in which the calculated products are added to the zero values ​​retrieved from the output register banks 20000, 20001, 20002, and 20003, respectively. SIMD0, SIMD1, SIMD2, and SIMD3 then write the accumulated values ​​to the addressed rows (row 4, row 11, row 0, and row 1, respectively) of the output register banks 20000, 20001, 20002, and 20003, respectively. Fig.29 In the illustrated embodiment, register select logic 2101 stores register select entries that enable routing values ​​from / to output register groups 20000 , 20001 , 20002 , and 20003 .

[0220] Notice, Fig.29 The multiply-accumulate operation implemented in advantageously uses the first column of the matrix I (i.e., the weight values ​​w 12,0 、w 45,0 and w 2,0 ) and the second column of matrix I (i.e., weight value w 7,1 ) both have non-zero weight values, thereby increasing efficiency (because no SIMD engine is idle, even if there are not enough non-zero entries in the first column of matrix I to supply all four SIMD engines).

[0221] like Fig.30 As illustrated, state machine and scheduler 431 then causes row 1 of operand A memory block 441 to be retrieved and loaded into operand A register file 411, and then passed to operand A buffer 416 within operand A allocation circuitry. Thus, operand A register file 411 and operand A allocation circuitry 416 store non-zero weight values ​​w for matrix I. 32,0 、w 21,1 、w 38,0 and w 23,2 .

[0222] The state machine and scheduler 431 also causes row 2 of operand B memory block 442 to be retrieved and loaded into operand B register file 412, and then passed to operand B buffer B2 within operand B allocation circuit 417. Thus, operand B register file 412 and operand B buffer B0 store activation values ​​d0, c0, b0, and a0 of matrix J, operand B register file 412 and operand B buffer B1 store activation values ​​d1, c1, b1, and a1, and operand B register file 412 operand B buffer B2 stores activation values ​​d2, c2, b2, and a2.

[0223] Note that the state machine and scheduler 431 retrieves activation values ​​from rows 0, 1, and 2 of operand B memory block 432, as these three activation values ​​are needed to calculate the required dot products associated with the retrieved weight values ​​(taken from the first three columns of matrix I) included in operand A.

[0224] The state machine and scheduler 431 causes the operand A allocation circuit 416 to convert the non-zero value w 32,0 、w 21,1 、w 38,0 and w 23,2The operand A register file 411 is routed to SIMD0, SIMD1, SIMD2, and SIMD3, respectively, as operand A. At the same time, the state machine and scheduler 431 causes the operand B allocation circuit 417 to route the values ​​d0, c0, b0, and a0 to each of SIMD0 and SIMD2 as operand B, causes the operand B allocation circuit 417 to route the values ​​d1, c1, b1, and a1 to SIMD1, and causes the operand B allocation circuit 417 to route the values ​​d2, c2, b2, and a2 to SIMD3.

[0225] SIMD0, SIMD1, SIMD2 and SIMD3 multiply the received operand A and operand B in the above manner. The state machine and scheduler 431 responds to the non-zero value w 32,0 、w 21,1 、w 38,0 and w 23,2 The previously determined rows in the associated output register banks 20000, 20001, 20002, and 20003 are addressed independently. That is, the state machine and scheduler 431 addresses row 8, row 5, row 9, and row 5 within the output register banks 20000, 20001, 20002, and 20003, respectively. As described above, all rows of the output register banks initially store a "0" value.

[0226] SIMD0, SIMD1, SIMD2, and SIMD3 perform an accumulation operation in which the calculated products are added to the zero values ​​retrieved from the output register banks 20000, 20001, 20002, and 20003, respectively. SIMD0, SIMD1, SIMD2, and SIMD3 then write the accumulated values ​​to the addressed rows (rows 8, 5, 9, and 5, respectively) of the output register banks 20000, 20001, 20002, and 20003, respectively.

[0227] Notice, Fig.30 The multiply-accumulate operation implemented in advantageously uses the first column of the matrix I (i.e., the weight values ​​w 32,0 and w 38,0 ), the second column of matrix I (i.e., weight value w 21,1 ) and the third column of matrix I (i.e., the weight value w 23,2 ), thereby increasing the efficiency of the operation (because no SIMD engine is idle).

[0228] like Fig.31As illustrated, state machine and scheduler 431 then causes row 2 of operand A memory block 441 to be retrieved and loaded into operand A register file 411, and then passed to an operand A buffer within operand A allocation circuit 416. Thus, operand A register file 411 and operand A allocation circuit 416 store non-zero weight values ​​w for matrix I. 8,2 、w 37,1 、w 10,2 and w 43,1 .

[0229] The activation values ​​already stored in the operand B buffers B1-B2 allocated for operand B are used to compare with the non-zero weight values ​​w of the matrix I 8,2 、w 37,1 、w 10,2 and w 43,1 Associative multiply-accumulate operation.

[0230] The state machine and scheduler 431 causes the operand A allocation circuit 416 to convert the non-zero value w 8,2 、w 37,1 、w 10,2 and w 43,1 The operand A register file 411 is routed to SIMD0, SIMD1, SIMD2, and SIMD3, respectively, as operand A. At the same time, the state machine and scheduler 431 causes the operand B allocation circuit 417 to route the values ​​d2, c2, b2, and a2 to each of SIMD0 and SIMD2 as operand B, and causes the operand B allocation circuit 417 to route the values ​​d1, c1, b1, and a1 to SIMD1 and SIMD3.

[0231] SIMD0, SIMD1, SIMD2 and SIMD3 multiply the received operand A and operand B in the above manner. The state machine and scheduler 431 responds to the non-zero value w 8,2 、w 37,1 、w 10,2 and w 43,1 The previously determined rows in the associated output register banks 20000, 20001, 20002, and 20003 are addressed independently. That is, the state machine and scheduler 431 addresses row 2, row 9, row 2, and row 10 within the output register banks 20000, 20001, 20002, and 20003, respectively. As described above, all rows of the output register banks initially store a "0" value.

[0232] SIMD0, SIMD1, SIMD2, and SIMD3 perform an accumulation operation in which the calculated products are added to the zero values ​​retrieved from the output register banks 20000, 20001, 20002, and 20003, respectively. SIMD0, SIMD1, SIMD2, and SIMD3 then write the accumulated values ​​to the addressed rows (row 2, row 9, row 2, and row 10, respectively) of the output register banks 20000, 20001, 20002, and 20003, respectively.

[0233] Notice, Fig.31 The multiply-accumulate operation implemented in advantageously uses the second column of the matrix I (i.e., the weight values ​​w 37,1 and w 43,1 ) and the third column of matrix I (i.e., the weight value w 8,2 and w 10,2 ) to increase the efficiency of the operation (because no SIMD engine is idle).

[0234] Although in Figure 29-Figure 31 The example of describes the processing of only three columns of the sparse matrix I, but it will be understood that the remaining columns of the sparse matrix I can be processed in the same manner as the first three columns. By packing the non-zero operand values ​​of the sparse matrix I in the above manner, the number of operations required to multiply the sparse matrix I and the matrix J is advantageously minimized.

[0235] Although operand packing logic 433 is shown as part of control logic 430 in the above embodiment, it will be understood that in alternative embodiments, the functionality of operand packing logic 433 can be implemented outside system 400. In such alternative embodiments, software can be used to identify non-zero values ​​of matrix I (because the weight values ​​of the network represented by the entries of matrix I are known), determine the output registers (and output register row addresses) associated with these non-zero values, identify the addresses of the values ​​of matrix J required to perform multiplication-accumulation operations on the non-zero values ​​of matrix I, and determine the manner in which the non-zero values ​​of matrix I should be packed in operand A register file 411. The method for performing these determinations is described in detail above. The packed operand A values ​​can then be loaded directly into operand A register file 411 (and / or system memory 440). The addresses required to load and access operand B register file 412 and the addresses required to access output registers 20000-20003 can be loaded into state machine and scheduler 431. The state machine and scheduler 431 then simply retrieves the non-zero value from memory and provides the required address signals during operation without any additional hardware complexity. In this way, this alternative embodiment advantageously reduces the hardware requirements of the system 400.

[0236] Although the present invention has been described in conjunction with several embodiments, it will be understood that the present invention is not limited to the disclosed embodiments but is capable of various modifications that will be apparent to those skilled in the art. Accordingly, the present invention is limited only by the appended claims.

Claims

1. A method of performing matrix multiplication of a first matrix and a second matrix using a computer system including N single instruction multiple data (SIMD) engines and N corresponding output register sets, wherein N is a number equal to or greater than 2, and wherein each of the output register sets includes a corresponding plurality of output registers, the method comprising: identifying a plurality of non-zero entries included in the first matrix, wherein each of the identified plurality of non-zero entries has a corresponding column address and a corresponding row address within the first matrix; for each of the identified plurality of non-zero entries, identifying a corresponding one of the N SIMD engines and a corresponding one of the N output registers to process the non-zero entry using the corresponding row address of the non-zero entry; sorting the identified plurality of non-zero entries to select N non-zero entries, each non-zero entry having a different identified corresponding one of the N output register sets; routing each of the selected N non-zero entries to an identified corresponding one of the N SIMD engines to perform a multiplication operation with an entry of the second matrix, wherein each of the N SIMD engines generates a plurality of products; for each of the selected N non-zero entries, identifying one of a plurality of output registers in an identified corresponding one of the N output register sets using a corresponding row address of the non-zero entry; as well as For each of the selected N non-zero entries, an accumulation operation is performed using an identified corresponding one of the N SIMD engines by accessing an identified one of the output registers within an identified corresponding one of the N output register groups.

2. The method according to claim 1, further comprising: For each of the N non-zero entries selected: identifying a corresponding row of entries within the second matrix in response to the corresponding column address of the non-zero entry; routing the identified corresponding entry row to the identified corresponding one of the N SIMD engines; as well as Each of the selected N non-zero entries is multiplied with the identified corresponding row of entries using the identified corresponding one of the N SIMD engines, thereby forming a corresponding plurality of products by the N SIMD engines.

3. The method according to claim 2, further comprising: For each of the selected N non-zero entries, a row within a corresponding one of the N output register groups is identified in response to the corresponding row address of the non-zero entry, wherein the identified row of the corresponding one of the N output register groups identifies a corresponding output register within the corresponding one of the N output register groups.

4. The method of claim 1, wherein each row of the first matrix represents a weight vector in a machine learning system, and each column of the second matrix represents an activation vector in the machine learning system.

5. The method according to claim 1, further comprising: Any zero entries in the first matrix are ignored.

6. The method according to claim 1, further comprising: A unique entry in the N output register sets is assigned to each dot product of the matrix multiplication.

7. The method of claim 1 , wherein each of the selected N non-zero entries is routed to an identified corresponding one of the N SIMD engines such that the identified corresponding one of the N SIMD engines receives multiple copies of the non-zero entry.

8. The method of claim 1, further comprising independently addressing each identified corresponding output register within a corresponding one of the N output register groups.

9. The method of claim 1, wherein 1 / 8 or less of the entries of the first matrix are non-zero entries.

10. The method of claim 1, wherein using a corresponding row address of the non-zero entry to identify a corresponding one of the N SIMD engines and a corresponding one of the N output register banks to process the non-zero entry comprises: The row address of the non-zero entry is divided by a number N, and a remainder of this division operation is then used to identify a corresponding one of the N SIMD engines and a corresponding one of the N output register sets.

11. The method of claim 3, wherein identifying a row within a corresponding one of the N output register banks in response to the corresponding row address of the non-zero entry comprises: The row addresses of the non-zero entries within the first matrix are divided by a number N and any remainders are ignored.

12. The method of claim 2, wherein identifying a corresponding entry row within the second matrix in response to the corresponding column address of the non-zero entry comprises: A row of entries in the second matrix having a row address equal to the corresponding column address of the non-zero entry is identified.

13. A computer system comprising: N single instruction multiple data (SIMD) engines, where N is a number equal to or greater than 2; as well as N corresponding output register groups, each corresponding output register group including a corresponding plurality of output registers, the computer system being configured to perform matrix multiplication of a first matrix and a second matrix by executing the following steps: identifying a plurality of non-zero entries included in the first matrix, wherein each of the identified plurality of non-zero entries has a corresponding column address and a corresponding row address within the first matrix; for each of the identified plurality of non-zero entries, using the corresponding row address of the non-zero entry to identify a corresponding one of the N SIMD engines and a corresponding one of the N output register sets to process the non-zero entry; sorting the identified plurality of non-zero entries to select N non-zero entries, each non-zero entry having a different identified corresponding one of the N output register sets; routing each of the selected N non-zero entries to an identified corresponding one of the N SIMD engines to perform a multiplication operation with an entry of the second matrix, wherein each of the N SIMD engines generates a plurality of products; for each of the selected N non-zero entries, identifying one of a plurality of output registers in an identified corresponding one of the N output register sets using a corresponding row address of the non-zero entry; as well as For each of the selected N non-zero entries, an accumulation operation is performed using an identified corresponding one of the N SIMD engines by accessing an identified one of the output registers within an identified corresponding one of the N output register groups.

14. The computer system of claim 13, further configured to perform the following steps: for each of the selected N non-zero entries, identifying a corresponding entry row within the second matrix in response to the corresponding column address of the non-zero entry; routing the identified corresponding entry row to an identified corresponding one of the N SIMD engines; and Each of the selected N non-zero entries is multiplied with the identified corresponding row of entries using the identified corresponding one of the N SIMD engines, thereby forming a corresponding plurality of products by the N SIMD engines.

15. The computer system of claim 13, further configured to perform the following steps: For each of the N non-zero entries selected, The corresponding row address responsive to the non-zero entry identifies a row within a corresponding one of the N output register sets, wherein the identified row of the respective one of the N output register sets identifies a corresponding output register within the corresponding one of the N output register sets.

16. The computer system of claim 13, wherein each row of the first matrix represents a weight vector in a machine learning system, and each column of the second matrix represents an activation vector in the machine learning system.

17. The computer system of claim 13, further configured to perform the step of: ignoring any zero entries in the first matrix.

18. The computer system of claim 13, further configured to perform the step of assigning a unique entry in the N output register sets to each dot product of the matrix multiplication.

19. The computer system of claim 13, wherein each of the selected N non-zero entries is routed to the identified corresponding one of the N SIMD engines such that the identified corresponding one of the N SIMD engines receives multiple copies of the non-zero entry.

20. The computer system of claim 13, further configured to perform the step of independently addressing each identified corresponding output register within a corresponding one of the N output register groups.

21. The computer system of claim 13, wherein 1 / 8 or less of the entries of the first matrix are non-zero entries.

22. The computer system of claim 13, wherein using the corresponding row address of the non-zero entry to identify a corresponding one of the N SIMD engines and a corresponding one of the N output register banks to process the non-zero entry comprises: The row address of the non-zero entry is divided by a number N, and a remainder of this division operation is then used to identify a corresponding one of the N SIMD engines and a corresponding one of the N output register sets.

23. The computer system of claim 15, wherein identifying a row within a corresponding one of the N output register sets in response to the corresponding row address of the non-zero entry comprises: The row addresses of the non-zero entries within the first matrix are divided by a number N and any remainders are ignored.

24. The computer system of claim 14, wherein identifying a corresponding entry row within the second matrix in response to the corresponding column address of the non-zero entry comprises: A row of entries in the second matrix having a row address equal to the corresponding column address of the non-zero entry is identified.

Images