Data compressor for matrix approximation for matrix multiplication operations

Abstract

A processing device is provided that includes a memory configured to store data and a processor. The processor includes a plurality of MACs configured to perform matrix multiplication of elements of a first matrix with elements of a second matrix. The processor also includes a plurality of logic devices configured to sum values ​​of bits of product exponent values ​​of elements of the first matrix and the second matrix and determine keep bit values ​​for product exponent values ​​that are to be kept for the matrix multiplication. The processor also includes a plurality of multiplexer arrays, each multiplexer array configured to receive bits of elements of the first matrix and the second matrix and the keep bit values, and to provide data for selecting which elements of the first matrix and the second matrix are provided to the MAC for the matrix multiplication.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of pending U.S. patent application Ser. No. 17 / 214,779, entitled "DATA COMPRESSOR FOR APPROXIMATION OF MATRICES FOR MATRIX MULTIPLY OPERATIONS," filed March 26, 2021, which is incorporated herein by reference in its entirety. This application is related to patent application Ser. No. 17 / 214,784, entitled "APPROXIMATION OF MATRICES FOR MATRIX MULTIPLY OPERATIONS," filed March 26, 2021, which is incorporated herein by reference as if fully set forth. [Background technology]

[0002] Matrix multiplication is an important building block across several application domains, including use in high-performance computing (HPC) and machine learning, and is used in convolutional neural networks, recurrent neural networks, and other forms of artificial neural networks.

[0003] Matrix multiplication techniques employ parallelization to improve the efficiency of matrix multiplication. For example, two matrices are typically divided into smaller parts (e.g., columns, rows, and portions of columns and rows), and a matrix multiplication operation of the two matrices is performed by performing multiple matrix multiplication calculations, each involving multiplication of a portion of one matrix with a portion of another matrix. The matrix multiplication calculations are mapped to and executed by different processor cores of a processor network to perform the matrix multiplication operation.

[0004] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram of an example device in which one or more features of the present disclosure may be implemented. [Figure 2] FIG. 1 is a block diagram illustrating exemplary components of a processor capable of implementing one or more features of the present disclosure. [Figure 3] 3 illustrates exemplary components of the data compressor and MAC shown in FIG. 2 for implementing one or more features of the present disclosure. [Figure 4] FIG. 1 is a flow diagram illustrating an example method for selecting data for matrix multiplication in accordance with aspects of the present disclosure. [Figure 5A] FIG. 2 illustrates a first example array of approximated product exponent values ​​in accordance with aspects of the present disclosure. [Figure 5B] FIG. 10 illustrates a second example array of approximated product exponent values ​​in accordance with aspects of the present disclosure. [Figure 5C] FIG. 10 illustrates a third example array of approximated product exponent values ​​in accordance with aspects of the present disclosure. [Figure 5D] FIG. 10 illustrates a fourth example array of approximated product exponent values ​​in accordance with aspects of the present disclosure. [Figure 6] FIG. 2 illustrates an example of sub-matrix multiplication of two matrices A and B with an output matrix C in accordance with an aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0006] As used herein, a program includes a sequence of instructions executed using one or more processors to perform a procedure or routine (e.g., an operation, calculation, function, process, job). The processing of programmed instructions and data includes one or more of a number of processing stages, such as, but not limited to, fetching, decoding, scheduling for execution, executing, and decoding the programmed instructions and data. Programmed instructions include, for example, applications and control programs, such as an operating system. A processor, for example, includes multiple processing cores (e.g., compute units (CUs)), each of which is configured to read and execute program instructions, such as instructions to perform matrix multiplication.

[0007] Matrix multiplication involves computing the dot product of subportions of data in a first matrix and a second matrix. A matrix multiplication operation involves the calculation C=A×B, where A, B, and C are matrices of size M×K, K×N, and M×N, respectively. Each element in matrix C is the dot product of a row of matrix A and a column of matrix B. For example, a multiply-accumulate operation computes the product of pairs of values, where each value corresponds to an element in a portion of a matrix (e.g., a row, a column, a portion of a row or column, or multiple rows or columns), and adds the product to an accumulator using a hardware component known as a multiplier-accumulator (MAC). For example, a 64×64 product can be implemented as four 16×16 MACs or eight 8×8 MACs. Matrix multiplication typically requires many calculations, which are time-consuming and expensive.

[0008] This application provides devices and methods for efficiently performing approximations of matrix multiplication. Features of this disclosure include data compression hardware configured to dynamically determine an output matrix by dropping some products (i.e., pairwise products of elements of two input matrices) from the products used for the dot product computation of a matrix multiplication of input matrices along a common dimension K of the two matrices. The dropped products are those approximated as having the smallest exponent sum among the products used for the dot product computation.

[0009] The data compression hardware includes keep logic and a set of multiplexer arrays. The keep logic is configured to determine keep bit values ​​based on the approximated product exponent values ​​and provide the keep bit values ​​to the set of multiplexer arrays to determine which elements in the arrays of data elements of the two matrices will be kept (i.e., not dropped) and provided to the MAC for matrix multiplication. The keep logic determines a target number of element values ​​to keep from the arrays of element values ​​of each matrix by summing the most significant bits (MSBs) of the product exponent values ​​and continuing through each corresponding set of next-most significant bits, summing bit values ​​of the same significance for each of the approximated product exponent values, and comparing the sum to a target number (e.g., 6). A feature of the present disclosure is the reduction of the number of products without first sorting the elements of the input arrays according to their values, which would otherwise be expensive to implement in hardware.

[0010] For example, a 64×64 product is reduced to a 48×48 product by retaining the 48 most approximated product values ​​among the 64×64 product values ​​(i.e., dropping the smallest 16 approximate values). As an example, if a 64×64 product is implemented as a 16×16 MAC, four 16×16 MACs are reduced to three 16×16 MACs, resulting in a 25% reduction in the time to perform the task and a reduction in the energy cost to perform the task. Similarly, if a 64×64 product is implemented as an 8×8 MAC, eight 8×8 MACs are reduced to six 8×8 MACs, also resulting in a 25% reduction in time. The target number of element values ​​to be retained and provided to the MAC to perform the matrix multiplication, or the target number of product values ​​to be dropped from the product value (i.e., dropped product values), may be any number and is determined based on various factors during runtime, such as the amount of result error that can be tolerated by the approximation for a particular task or application. For example, when used for machine learning training, the target number of products determined to be dropped is based on the impact of the approximation on the accuracy of the resulting network. The target number of product values ​​to be dropped may also be determined based on the size of the common dimension K. For example, based on a heuristic, additional product values ​​may be dropped for larger values ​​of K, and additional product values ​​may be kept and provided to the MAC for lower values ​​of K.

[0011] A feature of the present disclosure includes performing matrix multiplication on a variety of different data types, such as float data types (e.g., FP32, FP16, and BF16 formats) and integer data types (e.g., int8 format).

[0012] A processing device is provided that includes a memory configured to store data and a processor. The processor includes a plurality of MACs configured to perform matrix multiplication of elements of a first matrix and elements of a second matrix. The processor also includes a plurality of logic devices configured to sum values ​​of bits of product exponent values ​​of elements of the first matrix and the second matrix and determine keep bit values ​​for product exponent values ​​to be kept for the matrix multiplication. The processor also includes a plurality of multiplexer arrays, each multiplexer array configured to receive bits of elements of the first matrix and the second matrix and the keep bit values ​​and to provide data for selecting which elements of the first matrix and the second matrix are provided to the MAC for the matrix multiplication.

[0013] A processing device is provided that includes a memory configured to store data and a plurality of processor cores in communication with each other, each processor core including a plurality of MACs configured to perform matrix multiplication of elements of a first matrix with elements of a second matrix, a plurality of logic devices configured to sum values ​​of bits of approximated product exponents of elements of the first and second matrices to generate keep bit values, and a plurality of multiplexer arrays, each multiplexer array configured to receive the product exponents and the keep bit values ​​and to provide data for selecting which of the product exponents is provided to the MAC for matrix multiplication.

[0014] A processing device for use in matrix multiplication is provided, comprising a plurality of MACs configured to perform matrix multiplication of elements of a first matrix and elements of a second matrix. The processing device also comprises a plurality of logic devices configured to sum values ​​of bits of product exponent values ​​of elements of the first and second matrices and determine keep bit values ​​for product exponent values ​​to be kept for the matrix multiplication. The processing device further comprises a plurality of multiplexer arrays, each configured to receive bits of the product exponent values ​​and the keep bit values, and to provide data for selecting which of the product exponents to provide to the MAC for the matrix multiplication.

[0015] 1 is a block diagram of an example device 100 in which one or more features of the present disclosure may be implemented. Device 100 may include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. Device 100 includes a processor 102, a memory 104, a storage device 106, one or more input devices 108, and one or more output devices 110. Device 100 also optionally includes an input driver 112 and an output driver 114. It should be understood that device 100 may include additional components not shown in FIG. 1 .

[0016] In various alternatives, processor 102 includes any high-speed processing device, such as a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU. In various alternatives, memory 104 is located on the same die as processor 102 or is located separately from processor 102. Memory 104 includes volatile or non-volatile memory (e.g., random access memory (RAM), including dynamic RAM (DRAM) and static RAM (SRAM)). RAM includes, for example, cache memory, scratchpad memory, and registers.

[0017] The storage devices 106 include fixed or removable storage devices (e.g., hard disk drives, solid state drives, optical disks, flash drives). The input devices 108 include, but are not limited to, a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals). The output devices 110 include, but are not limited to, a display, a speaker, a printer, a haptic feedback device, one or more optics, an antenna, or a network connection (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals).

[0018] Input driver 112 communicates with processor 102 and input device 108, allowing processor 102 to receive input from input device 108. Output driver 114 communicates with processor 102 and output device 110, allowing processor 102 to send output to output device 110. Note that input driver 112 and output driver 114 are optional components, and that device 100 operates in the same manner without input driver 112 and output driver 114 present.

[0019] Features of the present disclosure are described herein using a CU as an example of a processor core. The CU includes one or more single instruction, multiple data (SIMD) units configured to perform operations at the request of the processor 102 in a parallel manner according to the SIMD paradigm. The SIMD paradigm allows multiple processing elements to share a single program control flow unit and program counter, thus executing the same program but with different data. In one example, each SIMD unit includes 16 lanes, each of which executes the same instruction simultaneously with other lanes in the SIMD unit but can execute the instruction with different data. Lanes can be predictively switched off when not all lanes need to execute a given instruction. Prediction can also be used to execute programs with branching control flow. More specifically, for programs with conditional branches or other instructions whose control flow is based on calculations performed by individual lanes, prediction of lanes corresponding to currently unexecuted control flow paths and serial execution of different control flow paths enables arbitrary control flow. The parallelism provided by the CU is well suited for matrix multiplications, such as those used in graphics-related operations such as pixel value calculations, vertex transformations, and other graphics operations.

[0020] 2 is a block diagram illustrating an example of the processor 102 shown in FIG. 1, which may implement one or more features of the present disclosure. As shown in FIG. 2, the processor 102 includes multiple CUs 202. Each CU 202 includes a level-N (e.g., level 1) cache controller 208 configured to communicate with and process data using a corresponding level-N cache 204. Alternatively, multiple level-N cache controllers 208 (e.g., a cache controller 208 for each CU 202, multiple cache controllers 208 each communicating with a group of CUs 202) are used to process data.

[0021] 2, processor 102 includes a level N+1 (e.g., level 2) cache controller 210 in communication with level N+1 cache 206 and is configured to process data using level N+1 cache 206. Alternatively, multiple level N+1 cache controllers in communication with one or more corresponding level N+1 caches may be used to process data. As shown in FIG. 2, cache controller 210 also communicates with the next cache level (e.g., level 3) and each level N cache controller 208. Additionally or alternatively, each CU 202 may communicate with a different type of memory 104, such as registers and scratchpad memory.

[0022] 2, each CU 202 includes a MAC 212 and a data compressor 214 in communication with the MAC 212. The MAC 212 is configured to calculate element-pair products (e.g., approximate product-exponent values) of portions of matrices and accumulate (e.g., store) the products to perform matrix multiplication operations.

[0023] For example, two matrices are typically divided into smaller portions (e.g., columns, rows, and portions of columns and rows), and a matrix multiplication operation of the two matrices is performed by performing multiple matrix multiplication calculations, each involving multiplication of a portion of one matrix with a portion of another matrix. The matrix multiplication calculations are mapped to and executed by different processor cores (e.g., CU 202) to perform the matrix multiplication operation.

[0024] An example of a partial matrix multiplication of two matrices A and B with an output matrix C is shown in FIG. 6. The matrix multiplication operation involves the calculation C=A×B, where A, B, and C are matrices of size M×K, K×N, and M×N, respectively. Each element of matrix C is the dot product of a row of matrix A and a column of matrix B. Matrices A and B are shown at the top of FIG. 6. For example, a portion of the matrix multiplication of matrices A and B is performed, in accordance with an aspect of the present disclosure, by approximating the product of four elements 602(1) through 602(4) of matrix B's element array 602 with four corresponding elements 604(1) through 604(4) of matrix B's array 604 to generate a result in portion 1 (P1) of matrix C.

[0025] In Figure 6, matrix A and matrix B are different sizes, but a matrix-matrix matrix multiplication operation can be performed as long as the number of columns in one matrix is ​​the same as the number of rows in the other. In the example shown in Figure 6, matrix A includes 8 columns and matrix B includes 8 rows. The numbers of columns and rows in matrices A and B are merely examples. Matrix multiplication, in accordance with features of the present disclosure, can be performed on matrices of any size that satisfy the general rules for matrix multiplication (e.g., the number of columns in matrix A is equal to the number of rows in matrix B).

[0026] 3, the data compressor 214 includes hardware circuitry configured to dynamically determine the output matrix C by dropping some products (i.e., pairwise products of elements of the two input matrices) from the products used for the dot product computation of the matrix multiplication of the input matrices along a common dimension K of the two matrices. The dropped products are approximated as those having the smallest exponent sum among the products used for the dot product computation.

[0027] Figure 3 illustrates exemplary components of the data compressor 214 and MAC 212 shown in Figure 2 for implementing one or more features of the present disclosure. As shown in Figure 3, the data compressor 214 includes a set of maintaining logic 306 of multiplexer arrays 300, each having six multiplexers. The number of multiplexers shown in Figure 3 is used as an example for selecting six elements from an array of eight data elements. Features of the present disclosure may include a different number of multiplexers depending on the size of the matrix portions and the number of target products to be maintained for matrix multiplication.

[0028] The maintain logic 306 includes logic circuits, such as fixed-function logic devices, arithmetic circuits, sequential logic devices (e.g., flip-flops, counters, and registers), and programmable logic devices, configured to perform different calculations on the approximated product values. The maintain logic 306 generates maintain signals that are provided to the set of multiplexer array 300 to determine which six elements from the array of eight data elements are maintained and provided to MAC 212 for matrix multiplication. For example, as described in more detail below, the maintain logic 306 is used to sum the values ​​of the corresponding significant bits of the product exponent values, starting with the most significant bit of the product exponent values, and compare each sum to a target number of product exponent values ​​(e.g., 6), until the target number of product exponent values ​​is determined.

[0029] 3 is configured to receive a bit of equal significance for each of the eight corresponding element values ​​(i.e., A0-A7) of matrix A and a keep signal (K0-K7) determined by keep logic 306. The output values ​​(e.g., A'0, A'1, A'2, A'3, A'4, and A'5) provided by multiplexer array 300 correspond to six of the eight elements selected to be provided to MAC 212 for matrix multiplication. The six elements selected to be kept are those determined to produce the six largest product values ​​of the eight product values ​​when multiplied with six corresponding elements from second matrix B.

[0030] The product value is approximated by 2 raised to the sum of the exponents of operands A and B. However, when MAC unit 212 performs exact multiplication, the mantissa values ​​are also multiplied and the value is rounded. As a result, one or more of the exact products of the dropped values ​​of A and B may be greater than the retained value, which may result in a small but acceptable reduction in the accuracy of the network for applications such as machine learning. Additionally, the potential for a small reduction in accuracy outweighs the reduction in time to perform the matrix multiplication operation (e.g., a 25% reduction in time as described above).

[0031] As shown in FIG. 3, N arrays are used to receive bits of equal significance for each of the eight element values ​​(A0-A7) of matrix A, where N is the number of bits in each element value. For example, if each of the element values ​​of matrix A contains four bits, four sets of multiplexer arrays 300 are used for matrix A. Each set of multiplexer arrays 300 receives, in parallel, a corresponding bit of element significance and a maintain signal from maintain logic 306. That is, a first set of multiplexer arrays 300 receives the most significant bit (i.e., the first bit) of the eight element values ​​of matrix A, a second set of multiplexer arrays 300 receives the next-most significant bit (i.e., the second bit) of the eight element values ​​of matrix A, a third set of multiplexer arrays 300 receives the third bit of the eight element values ​​of matrix A, and a fourth set of multiplexer arrays 300 receives the least significant bit of the eight element values ​​of matrix A.

[0032] Additionally, the data compressor 214 also includes a set of multiplexer arrays 300 configured to receive, in parallel, the element values ​​and corresponding bits of the significance of the maintain signal from the maintain logic 306. Thus, in an example where each of the element values ​​(B0-B7) includes four bits, four sets of multiplexer arrays 300 are also used for matrix B. The architecture of the set of multiplexer arrays 300 used for matrix B and the functionality of the set of multiplexer arrays 300 used for matrix B are the same as the set of multiplexer arrays 300 shown and described with respect to matrix A (except that the multiplexer arrays 300 used for matrix B receive the bits and maintain signals associated with matrix B). Therefore, a detailed description and illustration of the set of multiplexer arrays used for matrix B is unnecessary and will be omitted.

[0033] As shown in FIG. 3 , each set of multiplexer arrays 300 includes a first array of multiplexers 302 and a second array of multiplexers 304. The first array of multiplexers 302 includes six multiplexers 302(1), 302(2), 302(3), 302(4), 302(5), and 302(6). The second array of multiplexers 304 includes six multiplexers 304(1), 304(2), 304(3), 304(4), 304(5), and 304(6). The number of multiplexers in the first and second arrays 302 and 304 shown in FIG. 3 is used merely as an example when the eight input elements of a matrix are reduced to six. Features of the present disclosure may be implemented using multiplexer arrays having any number of multiplexers.

[0034] As shown in FIG. 3, each multiplexer 302(1)-302(6) in the first array 302 of the first set of multiplexer arrays 300 receives a bit value from the seventh element value (A6) of matrix A and a bit value from the eighth element value (A7) of matrix A. Each multiplexer 302(1)-302(6) in the first array 302 receives a corresponding U value (i.e., U0, U1, U2, U3, U4, U5), where U0=K6 (the maintain signal value for element value A6), U1=K0&&U0 (i.e., the value of U1=1 when K0 and K6 are both 1), U2=K1&&K0&&U0, U3=K2&&K1&&K0&&U0, U4=K3&&K2&&K1&&K0&&U0, and U5=K4&&K3&&K2&&K1&&K0&&U0.

[0035] Each multiplexer 304(1)-304(6) in the second array 304 receives a bit value from a corresponding element value of matrix A (A0-A5) and a corresponding keep bit value (i.e., K0-K5). Examples of the functionality of multiplexer arrays 302 and 304 are described below with respect to blocks 412-420 of FIG. 4.

[0036] 4 is a flow diagram illustrating an example method 400 for selecting data for matrix multiplication in accordance with aspects of the present disclosure. As described above, two matrices are typically divided into smaller portions (e.g., columns, rows, and portions of columns and rows), and a matrix multiplication operation of the two matrices is performed by performing multiple matrix multiplication calculations, each involving multiplication of a portion of one matrix with a portion of another matrix.

[0037] An example of a partial matrix multiplication of two matrices A and B with an output matrix C is shown in FIG. 6. The matrix multiplication operation involves the calculation C=A×B, where A, B, and C are matrices of size M×K, K×N, and M×N, respectively. Each element of matrix C is the dot product of a row of matrix A and a column of matrix B. Matrices A and B are shown at the top of FIG. 6. For example, a portion of the matrix multiplication of matrices A and B is performed, in accordance with an aspect of the present disclosure, by approximating the product of four elements 602(1) through 602(4) of matrix B's element array 602 with four corresponding elements 604(1) through 604(4) of matrix B's array 604 to generate a result in portion 1 (P1) of matrix C.

[0038] In Figure 6, matrix A and matrix B are different sizes, but a matrix-matrix matrix multiplication operation can be performed as long as the number of columns in one matrix is ​​the same as the number of rows in the other. In the example shown in Figure 6, matrix A includes 8 columns and matrix B includes 8 rows. The numbers of columns and rows in matrices A and B are merely examples. Matrix multiplication, in accordance with features of the present disclosure, can be performed on matrices of any size that satisfy the general rules for matrix multiplication (e.g., the number of columns in matrix A is equal to the number of rows in matrix B).

[0039] Referring again to FIG. 4 , as indicated at blocks 402 and 404, method 500 includes receiving (e.g., by data compressor 214 of CU 202) a portion of data for a first matrix A (DataA[8]) and a portion of data for a second matrix B (DataB[8]). In the example shown in FIG. 4 , each portion of data includes eight elements (i.e., values). That is, the eight elements of data for matrix A are represented in FIG. 4 as DataA[8], and the eight elements of data for matrix B are represented in FIG. 4 as DataB[8]. However, the number of elements shown in FIG. 4 is merely an example. The portions of data may include any number of elements that are provided to MAC 212 to perform the matrix multiplication.

[0040] The data portions of the first matrix and the second matrix may be any one of a number of data types, such as an integer data type (e.g., int8 format) and a float data type (e.g., BF16 format). The number of bits representing each element depends on the float format (e.g., FP32, FP16, BF16) and integer format (e.g., int8, int16, int32) used. For example, in the float format BF16, each element is represented by 16 bits, and each exponent is represented by 8 bits.

[0041] As shown in blocks 406 and 408, the method 300 includes extracting exponents from elements within the portion of data of the first matrix A and the second matrix B. That is, an exponent is extracted from each of the eight elements of the first matrix A (referred to as "E" in block 406). A [8] = Exponent Extraction(A[8])), the exponents are extracted from each of the eight elements of the second matrix B (denoted as “E [8] = Exponent Extraction(A[8])” in block 408). B [8] = Exponential Extraction (B[8])).

[0042] If the elements of the data portions of the first and second matrices are float data types, the exponent value can be simply extracted from the exponent bits of each element. For example, if the elements are in BF16 format, the first bit is the sign bit, bits 2 to 9 are exponent bits, and bits 10 to 16 are mantissa bits. Therefore, the exponent value can be extracted from the value of bits 2 to 9.

[0043] If the elements of the portions of data in the first matrix A and the second matrix B are integer data type (int8), then the exponent is the number of elements of each sub-portion of data (A a [8], B a [8]), and for each element, determine the number of leading zeros to drop, and represent each element as 1. M * 2 e (where M is the mantissa of the element value, and e is the exponent of the element value), and the exponent value of each element is set to [number of bits - 1] - L A(where L A is the number of leading zeros in the element).

[0044] For example, the first matrix (A a If the absolute value of the first element of [1]) is 00001011, it is determined that the four leading zeros are dropped. The element is 1.011 * 2 e The exponent value (e) for A[1] is expressed as [number of bits - 1] - L A =7-4=3. The second matrix (B a If the absolute value of the first element of [1]) is 00101111, it is determined that the two leading zeros are dropped. The element is 1.01111 * 2 e The exponent value (e) for A[1] is expressed as [number of bits - 1] - L B =7-2=5. Therefore, the exponent of the approximate product (P) of A[1] and B[1] is 14-(L A +L B )=14-8=6.

[0045] The same process is performed to find the approximate product of the remaining seven elements (A[2-8]) of the subportion of the first matrix A with the corresponding remaining seven elements (B[2-8]) of the subportion of the second matrix B.

[0046] At block 410, product values ​​(i.e., products) of eight corresponding pairs of elements of the first and second matrices are approximated. That is, the product of the first corresponding pair of elements is the sum of the extracted exponents of the first element of matrix A and the first element of matrix B (i.e., E A [1]+E B [1]), and the product of the second pair of elements is the sum of the extracted exponents of the second element of matrix A and the second element of matrix B (i.e., E A [2]+E B [2]), and the product of each of the eight corresponding pairs of elements is approximated (E P [8]=E A [8]+E B[8]), and so on.

[0047] As shown in block 412, the approximate product value E P [8] is analyzed using maintain logic 306, and based on the analysis, maintain signal K[8] is generated. Maintain signals K[0] through K[5] are provided to multiplexer 302, and maintain signal K[6] is provided to multiplexer 304 (where the value of maintain signal K7 is inferred from the values ​​of K0 through K6) to determine which six elements from both matrix A and matrix B are selected and provided to MAC 212 for matrix multiplication. Eight data elements (DataA[8]) and eight data elements (DataB[8]) from matrix A are again received, as shown in blocks 414 and 416. Six elements from matrix A (A'[6]) and six elements from matrix B (B'[6]) are selected from among the eight data elements provided to MAC 212 (i.e., dataSelect(A[8] and dataSelect(B[8]))) based on the outputs of multiplexer arrays 302 and 304, as shown in blocks 418 and 420.

[0048] Next, the approximate product value E PAn example of analyzing [8] and generating a keep signal (K[8]) will be described using four different example arrays of product exponent values ​​shown in FIGS. 5A-5D. For simplicity, the product exponent values ​​shown in FIGS. 5A-5D are four-bit values. However, features of the present disclosure may be implemented for product exponent values ​​having any number of bits. Furthermore, in the example described below, an array of eight product exponent values ​​is used, and a keep bit value is determined for each corresponding product exponent value. The keep logic 306 is configured to generate true values ​​(e.g., values ​​of 1) for six keep bits corresponding to six of the eight product exponent values ​​determined to be kept, and to generate false values ​​(e.g., values ​​of 0) for two keep bit values ​​corresponding to the remaining two target numbers of product exponent values ​​determined to be dropped. However, features of the present disclosure may be implemented for any number of product exponent values ​​and any target number of product exponent values ​​to be dropped. For example, if five of the eight product exponent values ​​of A and B are to be kept, the keep logic 306 is configured to generate true values ​​for the five keep bits corresponding to the five of the eight product exponent values ​​determined to be kept, and false values ​​are generated for the three keep bit values ​​corresponding to the remaining three target numbers of product exponent values ​​determined to be dropped.

[0049] For each of the arrays shown in Figures 5A through 5D, determining which keep signals are true and which keep values ​​are false involves summing the values ​​of the corresponding significance bits of the product exponent, starting with the most significant bit of the product exponent value, and comparing each sum to a target number of product exponent values ​​(e.g., 6 if 6 of the 8 values ​​of A and B are to be kept) until the target number of keep bit values ​​is determined to be true.

[0050] For example, for the array of eight product exponent values ​​shown in Figure 5A, the MSBs of the eight product exponent values ​​are summed. In this case, the sum of the MSBs of the eight product exponent values ​​is 6, which is equal to the target number of six product exponent values ​​to be retained. The six product exponent values ​​whose MSBs are 1 are determined as the product exponent values ​​to be retained, and the retain bit is set to 1 for the product exponent values ​​whose MSBs are 1. That is, the first product exponent value E p[0], the second product exponent value E p [1], the third product exponent value E p [2], the fifth product exponent value E p [4], the sixth product exponent value E p [5], the eighth product exponent value E p For [7], since the MSB is 1, the corresponding keep bit values ​​K0, K1, K2, K4, and K5 are set to 1, and the keep bit value of K7 is estimated to be 1 from the values ​​of K0 to K6. The two product exponent values ​​whose MSB is 0 are determined as product exponent values ​​to be dropped, and the keep bit values ​​K3 and K6 are set to 0.

[0051] For the array of eight product exponent values ​​shown in FIG. 5B, the sum of the MSBs of the eight product exponent values ​​is 8, which is greater than the target number of six values ​​of A and B to be retained. If the sum of the MSBs is greater than 6, the next least significant bits of the eight product exponent values ​​are summed. In this case, the sum of the next least significant bits (i.e., the second bits) of the eight product exponent values ​​starting with "11" is equal to 4. Therefore, the product exponent values ​​starting with "11" are identified as the four largest product exponent values, and the corresponding retain bit is set to 1 for the product exponent values ​​starting with "11". That is, the first product exponent value E p [0], the second product exponent value E p [1], the fourth product exponent value E p [3] and the eighth product exponent value E p Since each of [7] begins with "11," the corresponding keep bit values ​​K0, K1, and K3 are set to 1, and the keep bit value of K7 is inferred to be 1 from the values ​​of K0 through K6. However, since the four largest product exponent values ​​are 2 less than the target number 6, product exponent values ​​beginning with "101" are identified, and the next least significant bits (i.e., the third bits) of the product exponent values ​​beginning with "101" are summed, and the result is added to the previous sum. In this case, the sum of the third bits of the product exponent values ​​beginning with "101" equals 2, which, when added to the previous sum of 4, equals the target number 6. Therefore, the two product exponent values ​​beginning with "101" are also identified as product exponent values ​​to be kept. That is, the fifth product exponent value E p [4] and the sixth product exponent value E p[5] each start with "101", so the corresponding keep bit values ​​K4 and K5 are also set to 1, and the keep bit values ​​K2 and K6 corresponding to the remaining two product exponent values ​​to be dropped are set to 0.

[0052] For the array of eight product exponent values ​​shown in FIG. 5C , the sum of the MSBs of the eight product exponent values ​​is 8, which is greater than the target number of six values ​​in input matrices A and B to be retained. As described above, if the sum of the MSBs is greater than six, the next least significant bits of the eight product exponent values ​​are summed. In this case (as described above for the product exponent values ​​shown in FIG. 5B ), the sum of the next least significant bits (i.e., the second bits) of the eight product exponent values ​​equals four. Therefore, product exponent values ​​starting with “11” are identified as the four values ​​to be retained. Therefore, the keep bit values ​​for K0, K1, K3, and K7 are set to one for the product exponent values ​​starting with “11.” However, because the four largest product exponent values ​​are two less than the target number of six, the product exponent value starting with “101” is identified, and the next least significant bits (i.e., the third bits) of the product exponent values ​​starting with “101” are summed and added to the previous sum of four. In this case, the sum of the third bits of the product exponent values ​​beginning with "101" equals 3, which, when added to the previous sum of 4, equals 7. Because a sum of 7 is greater than the target number of 6, the least significant bits (i.e., the fourth bits) of the eight product exponent values ​​beginning with "1011" are summed, which again equals 3, which, when added to the previous sum of 4, again equals the target number of 6. In this case, because there are no more bits to add and the total number of previous sums is greater than the target number, the first two product exponent values ​​beginning with "1011" (E p [4] and E p [5]) are also kept. Therefore, the keep bit values ​​K4 and K5 are set to 1, and the keep bit values ​​K2 and K6 corresponding to the two remaining product exponent values ​​that are dropped are set to 0.

[0053] For the array of eight product exponent values ​​shown in Figure 5D, the sum of the MSBs of the eight product exponent values ​​is 8, which is greater than the target number of six product exponent values ​​to be retained (this is the same as described above for the product exponent values ​​shown in Figures 5B and 5C), except in this case each remaining bit is zero. In this case, the first six product exponent values ​​in the array are retained. That is, the retain bit is set to 1 for any of the target number of product exponent values ​​(e.g., the first six values ​​in the array).

[0054] An example of a hardware implementation of the maintain logic 306 for the Nth bit position (e.g., the position corresponding to the first, second, third, or fourth bit of the value shown in Figures 5-5C) is as follows: Data Input I i N , the i-th data input to the N-th bit, i=0,1,...7 Total N =Sum(I i N &&Q i N+1 ) Accumulated SumA N =C N+1 +(D N+1 ?0:T N ) Enable N =(T N !=0)&&(A N <=(8-2)) / / Comment: Two values ​​of 8 are dropped Invert V N ={(T N !=0)&&(A N <(8-2))}||(T N ==0) QualityQ i N =Q i N+1 &&(V N ?!I i N :I i N )

[0055] An example of a hardware implementation of the maintain logic 306 for selecting the bit in the Nth position is as follows: Done D N =D N+1 Count C N =C N+1 For i=0 to 7 { KeepK i N =I i N &&Q i N+1 &&E N &&!D N C N =C N +K i N D N =(C N ==(8-2)) }

[0056] An example of a hardware implementation of the maintain logic 306 to select the bit in the −1 th position (the bit position to the right of the least significant bit in FIGS. 5C and 5D) is as follows: Done D N =D N +1 Count C N =C N +1 When I=0 to 7 { K i N =Q i N+1 &&!D N C N =C N +K i N D N =(C N ==(8-2)) } Keep i=ΣK iN where sum represents an OR over all values ​​of K

[0057] A matrix multiplication is then performed on six selected data elements from matrix A and six selected data elements from matrix B. In addition, information produced by the matrix multiplication operation may be displayed on a display device (e.g., output device 110 of FIG. 1).

[0058] It should be understood that many variations are possible based on the disclosure herein, and although features and elements are described above in particular combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0059] The provided methods can be implemented in a general-purpose computer, processor, or processor core. Suitable processors include, by way of example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine. Such processors can be fabricated by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data (instructions that can be stored on a computer-readable medium), including a netlist. The result of such processing can be a mask work that is then used in a semiconductor manufacturing process to produce a processor implementing features of the present disclosure.

[0060] The methods or flow diagrams provided herein may be implemented in a computer program, software, or firmware embodied in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims

1. 1. A processing device, comprising: a memory configured to store data; a processor, The processor: a plurality of multiplier-accumulators (MACs) configured to perform matrix multiplication of elements of a first matrix with elements of a second matrix; a plurality of logic devices configured to sum values ​​of bits of product exponent values ​​of elements of the first matrix and the second matrix to determine retained bit values ​​for product exponent values ​​that are retained for matrix multiplication; a plurality of multiplexer arrays; Each of the plurality of multiplexer arrays comprises: receiving bits of elements of the first matrix and the second matrix and the preserve bit value; providing data for selecting which elements of the first matrix and the second matrix are provided to the MAC for matrix multiplication; configured to: Processing device.

2. the plurality of logic devices are configured to sum the values ​​of corresponding significance bits of the product exponent values; The processing device of claim 1 .

3. the plurality of logic devices are configured to sum the values ​​of the most significant bits of the product exponent values ​​by starting with summing the most significant bits and comparing each of the sums to a target number of elements of the first matrix and the second matrix maintained and provided to the MAC for matrix multiplication; The processing device of claim 2.

4. a target number of elements of the first matrix and the second matrix is ​​determined to be maintained for matrix multiplication; the plurality of logic devices are configured to determine which of the target numbers of elements of the first matrix and the second matrix to maintain when the remaining bits of the product exponent value are zero. The processing device of claim 2.

5. Each of the plurality of multiplexer arrays comprises: a first array of multiplexers, each multiplexer comprising: any two elements of the first matrix and the second matrix; a selection value for selecting which of the two elements to output; a first array of multiplexers configured to receive a second array of multiplexers, each multiplexer comprising: remaining elements of either the first matrix or the second matrix that are different from the two elements; any of the keep bit values ​​for selecting whether to output an element output from any of the multiplexers in the first array or the remaining elements; and a second array of multiplexers configured to receive The processing device of claim 1 .

6. the number of multiplexer arrays is based on the number of bits representing element values ​​of the first matrix and the second matrix; The processing device of claim 1 .

7. the multiplexer array comprises a first set of arrays configured to receive elements of the first matrix and a second set of arrays configured to receive elements of the second matrix; The processing device of claim 1 .

8. Further comprising a display device, the information generated from the matrix multiplication is displayed on the display device. The processing device of claim 1 .

9. 1. A processing device, comprising: a memory configured to store data; a plurality of processor cores in communication with each other; Each of the plurality of processor cores a plurality of multiplier-accumulators (MACs) configured to perform matrix multiplication of elements of a first matrix with elements of a second matrix; a plurality of logic devices configured to sum values ​​of bits of product exponent values ​​of elements of the first matrix and the second matrix to determine retained bit values ​​for product exponent values ​​that are retained for matrix multiplication; a plurality of multiplexer arrays; Each of the plurality of multiplexer arrays comprises: receiving bits of elements of the first matrix and the second matrix and the preserve bit value; providing data for selecting which elements of the first matrix and the second matrix are provided to the MAC for matrix multiplication; configured to: Processing device.

10. the plurality of logic devices are configured to sum the values ​​of corresponding significance bits of the product exponent values; The processing device of claim 9.

11. the plurality of logic devices are configured to sum the values ​​of the most significant bits of the product exponent values ​​by starting with summing the most significant bits and comparing each of the sums to a target number of elements of the first matrix and the second matrix maintained and provided to the MAC for matrix multiplication; The processing device of claim 10.

12. a target number of elements of the first matrix and the second matrix is ​​determined to be maintained for matrix multiplication; the plurality of logic devices are configured to determine which of the target numbers of elements of the first matrix and the second matrix to maintain when the remaining bits of the product exponent value are zero. The processing device of claim 10.

13. Each of the plurality of multiplexer arrays comprises: a first array of multiplexers, each multiplexer comprising: any two elements of the first matrix and the second matrix; a selection value for selecting which of the two elements to output; a first array of multiplexers configured to receive a second array of multiplexers, each multiplexer comprising: remaining elements of either the first matrix or the second matrix that are different from the two elements; any of the keep bit values ​​for selecting whether to output an element output from any of the multiplexers in the first array or the remaining elements; and a second array of multiplexers configured to receive The processing device of claim 9.

14. the number of multiplexer arrays is based on the number of bits representing element values ​​of the first matrix and the second matrix; The processing device of claim 9.

15. the multiplexer array comprises a first set of arrays configured to receive the product exponent values ​​for elements of the first matrix and a second set of arrays configured to receive the product exponent values ​​for elements of the second matrix; The processing device of claim 9.

16. 1. A processing device for use with matrix multiplication, comprising: a plurality of multiplier-accumulators (MACs) configured to perform matrix multiplication of elements of a first matrix with elements of a second matrix; a plurality of logic devices configured to sum values ​​of bits of product exponent values ​​of elements of the first matrix and the second matrix to determine retained bit values ​​for product exponent values ​​that are retained for matrix multiplication; a plurality of multiplexer arrays; Each of the plurality of multiplexer arrays comprises: receiving bits of elements of the first matrix and the second matrix and the preserve bit value; providing data for selecting which elements of the first matrix and the second matrix are provided to the MAC for matrix multiplication; configured to: Processing device.

17. the plurality of logic devices are configured to sum the values ​​of corresponding significance bits of the product exponent values; 17. The processing device of claim 16.

18. the plurality of logic devices are configured to sum the values ​​of the most significant bits of the product exponent values ​​by starting with summing the most significant bits and comparing each of the sums to a target number of elements of the first matrix and the second matrix maintained and provided to the MAC for matrix multiplication; 20. The processing device of claim 17.

19. a target number of elements of the first matrix and the second matrix is ​​determined to be maintained for matrix multiplication; the plurality of logic devices are configured to determine which of the target numbers of elements of the first matrix and the second matrix to maintain when the remaining bits of the product exponent value are zero.

20. The processing device of claim 17.

20. Each of the plurality of multiplexer arrays comprises: a first array of multiplexers, each multiplexer comprising: any two elements of the first matrix and the second matrix; a selection value for selecting which of the two elements to output; a first array of multiplexers configured to receive a second array of multiplexers, each multiplexer comprising: remaining elements of either the first matrix or the second matrix that are different from the two elements; any of the keep bit values ​​for selecting whether to output an element output from any of the multiplexers in the first array or the remaining elements; and a second array of multiplexers configured to receive 17. The processing device of claim 16.

Images