Improved Multiplier Circuit

Abstract

To provide a system and a method which are improved in order to implement matrix multiplication.SOLUTION: A system performs matrix multiplication of a vector with a two-dimensional matrix by evaluating whether or not the vector includes zero values. Rows of the matrix are loaded into a first memory device from a second device. Rows corresponding to indexes of the zero values are not loaded. A dot product for columns of the matrix and an input vector is performed and stored. The matrix may be stored in the second memory device such that only entries for non-zero entries are stored. The rows of the matrix may be reconstructed in the first memory device from these entries.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] [Related Applications] This application is related to the following applications, all of which are hereby incorporated by reference in their entirety: U.S. Patent Application No. 62 / 373,518, filed August 11, 2016, entitled “Zero Coefficient Skipping Convolution Neural Network Engine” (Attorney Docket No. GQLA-03800); U.S. Patent Application No. 15 / 671,829, filed August 8, 2017, entitled "Zero Coefficient Skipping Convolution Neural Network Engine" (Attorney Docket No. GQLA-03801); and U.S. Patent Application No. 15 / 671,860 (Attorney Docket No. GQLA-03802), filed August 8, 2017, and entitled “Zero Coefficient Skipping Convolution Neural Network Engine.”

[0002] [Field of the Invention] The present invention relates to a system and method for performing matrix multiplication, dot products, or other multiply-accumulate operations. [Background technology]

[0003] Many graphics processing applications rely on matrix multiplication in one form or another. Matrix multiplication is also part of many mathematical problems that participate in computational solutions. In many applications of matrix multiplication, particularly to graphics processing and artificial intelligence, the matrices involved can be very large, such as on the order of thousands of rows and columns.

[0004] The systems and methods disclosed herein provide an improved approach for implementing matrix multiplication using computing devices. [Brief explanation of the drawings]

[0005] In order that the advantages of the invention may be readily understood, a more particular description of the invention briefly described above will be given by reference to specific embodiments which are illustrated in the accompanying drawings. The invention will be described and explained with additional particularity and detail through the use of the accompanying drawings, with the understanding that these drawings depict only typical embodiments of the invention and therefore should not be considered limiting of its scope. [Figure 1] FIG. 1 is a diagram of a computing device in which the systems and methods disclosed herein may be implemented. [Figure 2] An exemplary application for the methods disclosed herein will now be described. [Figure 3] FIG. 1 illustrates an improved approach for matrix multiplication according to an embodiment of the present invention. [Figure 4] FIG. 2 is a process flow diagram of a method for performing matrix multiplication according to an embodiment of the present invention. [Figure 5] FIG. 1 is a schematic block diagram of an alternative architecture for implementing matrix multiplication in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0006] It will be readily understood that the components of the present invention, as generally described and illustrated in the Figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of embodiments of the invention, as represented in the Figures, is not intended to limit the scope of the invention as claimed, but is merely representative of some examples of embodiments presently contemplated in accordance with the invention. The presently described embodiments will be best understood by reference to the drawings, wherein like parts are designated with like numerals throughout.

[0007] Embodiments in accordance with the present invention may be embodied as an apparatus, a method, or a computer program product. Accordingly, the present invention may be embodied in an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or The present invention may take the form of an embodiment combining software and hardware aspects, all of which may be generically referred to herein as a “module” or a “system.” Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0008] Any combination of one or more computer usable or computer readable media, including non-transitory media, may be utilized. For example, computer readable media may include one or more of portable computer disks, hard disks, random access memory (RAM) devices, read-only memory (ROM) devices, erasable programmable read-only memory (EPROM or flash memory) devices, portable compact disk read-only memories (CDROMs), optical storage devices, and magnetic storage devices. In selected embodiments, computer readable media may include any non-transitory medium that may contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0009] Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, or C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may be executed entirely on a computer system as a standalone software package, on a standalone hardware unit, partially on a remote computer some distance from the computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider).

[0010] The present invention is described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions or code. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, to produce a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0011] These computer program instructions may also be stored on a non-transitory computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture that includes instruction means that implement the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0012] Computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps performed on the computer or other programmable device to produce a computer-implemented process, such that instructions executing on the computer or other programmable device provide a process for implementing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0013] 1 is a block diagram illustrating an exemplary computing device 100. Computing device 100 may be used to perform various procedures, such as those discussed herein. Computing device 100 may function as a server, a client, or any other computing entity. The computing device may incorporate circuitry to perform the methods disclosed herein and may execute one or more application programs, such as an application program that invokes the methods disclosed herein to calculate trigonometric functions. Computing device 100 may be any of a wide variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, and a tablet computer.

[0014] Computing device 100 includes one or more processors 102, one or more memory devices 104, one or more interfaces 106, one or more mass storage devices 108, one or more input / output (I / O) devices 110, and a display device 130, all coupled to a bus 112. Processor 102 includes one or more processors or controllers that execute instructions stored on memory device(s) 104 and / or mass storage device(s) 108. Processor 102 may also include various types of computer-readable media, such as cache memory.

[0015] The memory device 104 includes a variety of computer-readable media, such as volatile memory (e.g., random access memory (RAM) 114) and / or non-volatile memory (e.g., read-only memory (ROM) 116). The memory device 104 may also include rewritable ROM, such as flash memory.

[0016] Mass storage device 108 includes a variety of computer-readable media, such as magnetic tape, magnetic disks, optical disks, and solid-state memory (e.g., flash memory). As shown in Figure 1, a particular mass storage device is a hard disk drive 124. Mass storage device 108 may also include a variety of drives to enable reading from and / or writing to a variety of computer-readable media. Mass storage device 108 includes removable media 126 and / or non-removable media.

[0017] I / O devices 110 include various devices that allow data and / or other information to be entered into and / or retrieved from computing device 100. Exemplary I / O devices 110 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCD or other image capture devices, and the like.

[0018] Display device 130 includes any type of device capable of displaying information to one or more users of computing device 100. Examples of display device 130 include monitors, display terminals, video projection devices, and the like.

[0019] A graphics processing unit (GPU) 132 may be coupled to processor 102 and / or display device 130. The GPU may be operable to render computer-generated images and perform other graphics processing. The GPU may include some or all of the functionality of a general-purpose processor, such as processor 102. The GPU may also include additional functionality specific to graphics processing. The GPU may include hard-coded and / or hard-wired graphics functionality related to coordinate transformations, shading, texturing, rasterization, and other functions useful for rendering computer-generated images.

[0020] The interface 106 includes various interfaces that allow the computing device 100 to interact with other systems, devices, or computing environments. An exemplary interface 106 includes any number of different network interfaces 120, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interfaces include a user interface 118 and a peripheral device interface 122. The interface 106 may also include one or more user interface elements 118. The interface 106 may also include one or more peripheral interfaces, such as interfaces for a printer, a pointing device (mouse, trackpad, etc.), a keyboard, etc.

[0021] Bus 112 allows processor 102, memory device 104, interface 106, mass storage device 108, and I / O device 110 to communicate with each other, along with other devices or components coupled to bus 112. Bus 112 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, and a USB bus.

[0022] In some embodiments, processor 102 may include a cache 134, such as one or both of an L1 cache and an L2 cache. GPU 132 may also include a cache 136, which may also include one or both of an L1 cache and an L2 cache. Processor 102 and GPU 132 may include circuitry or otherwise be programmed to perform matrix multiplication using caches 134, 136, as described below.

[0023] For purposes of explanation, programs and other executable program components are illustrated herein as separate blocks, but it is understood that such programs and components may reside at various times in different storage components of computing device 100 and be executed by processor 102. Alternatively, the systems and procedures described herein may be implemented in hardware, or in a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) may be programmed to execute one or more of the systems and procedures described herein.

[0024] 2, vector A may contain one or more elements that are zero. Vector A may be a row of pixels from an image, a set of samples of an audio signal, or any other vector of values that can undergo matrix multiplication.

[0025] 3, vector A may be multiplied with matrix M, i.e., a two-dimensional array of values. In the illustrated example, matrix M includes a row dimension j and a column dimension k. To obtain the product, each column 302 of matrix M may be multiplied with vector A, such as by performing a dot product operation in which each element at a vector index in vector A is multiplied with the element in the column 302 having a row index equal to the vector index. The products for all of the vector indices are then summed to obtain the final result, which includes an output vector V having a number of elements equal to the number of columns in M.

[0026] 2 and 3 illustrate an example of an operation that may be improved according to the methods disclosed herein. In some embodiments, vector A may be a set of samples from an audio signal or other sampled signal. In particular, the methods disclosed herein may be applied to any application in which a vector is multiplied by a matrix. The methods disclosed herein are particularly useful for very large vectors and matrices, e.g., vectors with 1000 or more elements and matrices with 1000 or more rows.

[0027] As shown in FIG. 3, one or more elements of vector A may be zero, indicated by the stippled elements. In some applications, elements with zero values may be common, especially in input vector A for machine learning applications such as neural networks. As a result, the stippled rows of matrix M will have no effect on the matrix multiplication. In many applications, the time spent retrieving matrix M from memory is a significant source of latency in matrix multiplication. Therefore, as described in more detail below, loading rows with indices corresponding to zero-valued elements of vector A is omitted.

[0028] 4 illustrates an example method 400 for performing matrix multiplication in a manner that accounts for zero-valued elements of vector A. Method 400 may be performed by processor 102, GPU 132, or other processing device.

[0029] Method 400 may include loading 402 an activation vector A from a second memory device into a first memory device. For example, the second memory device may be RAM 114, and the first memory device may be cache 134, 136. In other embodiments, the second memory device is mass storage device 108, and the first memory device is 134, 136, or RAM 114. In other embodiments, the first memory device is an L1 cache, and the second memory is an L2 cache.

[0030] An index i is initialized to zero (step 404). Index i corresponds to an element of activation vector A, and the corresponding column index j=i of matrix M that may be multiplied with that element. Step 404 may include initializing a set of accumulators R(k) to be zero, the number of accumulators being equal to the number of columns of M, such that R(k=j) corresponds to column j of M.

[0031] If element A(i) is found not to be equal to zero (406), row j=i of matrix M is loaded from the second memory device into the first memory device (408).

[0032] In some embodiments, the matrix M may be compressed, such that step 408 may include decompressing row j of the matrix M and storing the decompressed row in a first memory device.

[0033] In some embodiments, matrix M may be stored in the form of multiple entries, with each entry being of the format <row index><column index><element value>. In such a manner, the space required to store the sparse matrix is reduced. Thus, loading in step 406 may include either (a) loading these entries into a first memory device. For example, locations within a line of memory corresponding to the column index of an element in a row may be populated with data corresponding to that column index and row j. Rows of matrix M may be individually compressed so that individual rows can be decompressed and loaded. As is apparent from FIG. 4, if A(i)=0, then no attempt will be made to load row j=i of matrix M.

[0034] Method 400 may include processing (410), either in parallel or sequentially, each element M(j,k), for k=0 to kMax, where kMax is equal to the number of columns in M minus 1. In particular, for each value of k, an accumulator R(k) is set equal to the current value of R(k) + A(i) * M(i,k) in step 412. If processing (410) is done sequentially, step 412 may be omitted for values of k for which M(i,k) is equal to zero. In some embodiments, if processing (410) is done in parallel, step 412 is also omitted for those values of k for which M(i,k) is zero. If processing (410) is done in parallel, an array of multiplicative accumulators may be used, one for each column k, such that step 412 is performed by one of the multiplicative accumulators for each value of k.

[0035] If, following processing according to step 410, A(i) is found to be zero, method 400 may include evaluating whether i is equal to iMax (the number of elements in A minus 1). If not, i is incremented in step 414 and processing continues in step 406. If so, the method ends and the value in accumulator R(k) is a vector representing the result of matrix multiplication A·M, as shown in FIG.

[0036] Referring to FIG. 5, an illustrated architecture 500 and method 600 can be used to perform matrix multiplication such that the column-wise multiplication of C is parallelized.

[0037] As shown in FIG. 5, activation vector A and matrix M may be initially stored in second memory 501. Loader 502 loads some or all of A and M into first memory 504. Loader 502 may include a zero checker 506 that evaluates the value of A as part of the loading process (see step 408 of FIG. 4). Loader 502 may store A as A' in first memory 504 such that A' does not contain any elements with a value of zero. Thus, if A has B elements with a value of zero, A' will have only N B elements. If M is compressed, loader 502 may decompress the rows of M that are retrieved. As described above for method 400, only those rows of M that do not correspond to the zero index of A are loaded.

[0038] The element value for each entry in the row being loaded is stored in the column position indicated by the <column> index field for that entry. In other embodiments, the entries for a row are stored in the format <row index><column index><element value>.

[0039] Sequencer 508 may manage the performance of the matrix multiplication, for example, by loading an element of A′ as a first argument of an array of multipliers 510. Sequencer 508 may load a row of M as a second argument of the array of multipliers 510 as part of performing step 412 for that row. The array of multipliers 510 performs the multiplication, and the products are input to accumulators 512, i.e., the accumulators for each of M, which add the outputs of the multipliers to their current values. Sequencer 508 may instruct which multipliers to output to sum only the outputs of multipliers that have elements of A′ loaded as their first argument and only the outputs of multipliers that have non-zero elements of M loaded as their second argument. The outputs of accumulators 512 may be written to output buffer 514 and may be used according to any method known in the art for using matrix multiplication.

[0040] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

[0041] The claims are listed below:

Claims

1. A multiplier array; an accumulator; a first memory; a second memory having a larger capacity and a slower access time than the first memory; a plurality of components including a loader and a sequencer; The loader identifying one or more zero-indexed positions within the input vector such that the value of the input vector at said one or more zero-indexed positions is zero; loading a portion of a two-dimensional array defining a first dimension and a second dimension from the second memory into the first memory; the portion includes zero and non-zero values and does not include values of the two-dimensional array having a first index in the first dimension that corresponds to the one or more zero-index positions; The sequencer managing the performance of matrix multiplication by sequentially loading each non-zero element of the input vector and each row of the portion of the two-dimensional array into the multiplier array; directing the accumulator to add the output of the multiplier array only to non-zero values in each row of the portion of the two-dimensional array; system.

2. The sequencer, for the two-dimensional array designated as M and the input vector designated as A, setting i=0, performing (a) and (b) until i is equal to the dimension of the two-dimensional array in the second dimension, and setting an accumulator R(k)=0 for k=0 to (kMax-1), where kMax is the number of elements of the two-dimensional array in the second dimension, and (a) and (b) (a) if A(i) is not equal to zero, then computing R(k) for k=0 to (kMax-1) as R(k)=R(k)+A(i)*M(i,k); (b) incrementing i by 1; managing the performance of the matrix multiplication by including The system of claim 1 .

3. the loader is further programmed to obtain the input vector by retrieving a row of pixels from an image of a set of images; The system of claim 1 .

4. the loader is further programmed to obtain the input vector as a set of samples from an audio signal. The system of claim 1 .

5. the loader is further programmed to obtain the portion of the two-dimensional array from a plurality of entries; each entry including the first index and a second index to a non-zero value in the two-dimensional array; The system of claim 1 .

6. the loader is further programmed to obtain the plurality of entries by decompressing the portion of the two-dimensional array. The system of claim 5.

7. the first memory is a cache on a chip that includes a processing device; the second memory is a random access memory (RAM); The system of claim 1 .

Images