Improved multiplier circuit

Abstract

This invention relates to an improved multiplier circuit. A system performs matrix multiplication of a vector with a two-dimensional matrix by evaluating whether the vector contains zero values. The rows of the matrix are loaded from a second device into a first storage device. Rows corresponding to zero-value indices are not loaded. The dot product of the columns of the matrix with the input vector is performed and stored. The matrix can be stored in the second storage device such that only non-zero entries are stored. The rows of the matrix can be reconstructed from these entries stored in the first storage device.

Description

[0001] Related applications

[0002] This application relates to the following applications, the contents of which are incorporated herein by reference:

[0003] U.S. patent application No. 62 / 373,518 entitled “Zero Coefficient Skip Convolutional Neural Network Engine”, filed on August 11, 2016;

[0004] U.S. patent application No. 15 / 671,829, entitled "Zero-coefficient skipping convolutional neural network engine," filed on August 8, 2017; and

[0005] U.S. patent application No. 15 / 671,860 entitled “Zero Coefficient Skip Convolutional Neural Network Engine”, filed on August 8, 2017. Technical Field

[0006] This invention relates to a system and method for performing matrix multiplication, dot product, or other multiplication / accumulation operations. Background Technology

[0007] Many graphics processing applications rely on some form of matrix multiplication. Matrix multiplication is also part of many mathematical problems that are often suited to computational solutions. Many applications of matrix multiplication, particularly in graphics processing and artificial intelligence, involve matrices that can be extremely large, with tens of thousands of rows and columns.

[0008] The system and method disclosed in this paper provide an improved approach for implementing matrix multiplication on computing devices. Attached Figure Description

[0009] To facilitate understanding of the advantages of the present invention, this application will provide a more detailed description of the invention, which has been briefly described, with reference to specific embodiments shown in the accompanying drawings. These drawings depict only typical embodiments of the invention and should not be considered as limiting the scope of the invention. This application will describe and explain the invention with additional features and details using the accompanying drawings.

[0010] Figure 1 A computing device is shown that can implement the systems and methods disclosed herein;

[0011] Figure 2 An example application of the method disclosed in this paper is shown.

[0012] Figure 3 A schematic diagram of an improved method for matrix multiplication according to an embodiment of the present invention is provided.

[0013] Figure 4A flowchart of a method for performing matrix multiplication according to one embodiment of the present application is provided.

[0014] Figure 5 is a schematic block diagram of an alternative architecture for performing matrix multiplication according to one embodiment of the present application. DETAILED DESCRIPTION

[0015] The components of the application can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the application, as represented in the drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of the basic concepts of the application. The scope of the application is set forth with particularity in the appended claims. The application will be better understood from the detailed description when considered in connection with the following drawings, in which like symbols indicate similar components, wherein:

[0016] Embodiments according to the present application can be implemented as a device, a method or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "module" or "system." Furthermore, the present application can take the form of a computer program product on any tangible medium of expression having computer-usable program code embodied in the medium.

[0017] Any combination of one or more computer-usable or computer-readable media can be utilized. The computer- readable media can include a computer-readable storage medium having stored thereon computer-executable program code that, when implemented, embarks upon carrying out processes and / or devices determined according to the instructions of the program code. The computer- readable storage medium can include a floppy disk, a floppy disk drive, a hard disk, a solid-state drive, a magnetic tape, a hard disk drive, a magnetic disk drive, an optical disk drive, a CD-ROM drive, a CD-R drive, a CD-RW drive, a DVD-ROM drive, a DVD-R drive, a DVD-RW drive, a DVD-RAM drive, a flash memory device, a phase-change memory device, a RAM device, a ROM device, any suitable memory device, any suitable medium that stores the desired information and / or instructions, or any suitable combination of the foregoing. Computer-readable media can include, at least in part, a computer-readable storage medium.

[0018] The computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on a computer system, in a stand-alone software package, on a hardware unit of a remote computer, or on a remote computer or server across a network, such as a local area network (LAN) or a wide area network (WAN). In the latter scenario, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0019] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0020] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0021] Figure 1 A block diagram of an example computing device 100 is shown. The computing device 100 can be used to perform various processes, such as the processes discussed herein. The computing device 100 can be used as a server, a client, or any other computing entity. The computing device 100 can contain circuitry for executing the methods disclosed herein and can execute one or more application programs, such as an application program that calls the methods disclosed herein to compute trigonometric functions. The computing device 100 can be any of a wide variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, a tablet computer, and the like.

[0022] Computing device 100 includes one or more processors 102, one or more storage devices 104, one or more interfaces 106, one or more mass storage devices 108, one or more input / output (I / O) devices 110, and a display device 130, all coupled to bus 112. Processor 102 includes one or more processors or controllers that execute instructions stored in storage device 104 and / or memory. Processor 102 can also include various types of computer-readable media, such as cache memory.

[0023] Storage device 104 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 114) and / or non-volatile memory (e.g., read only memory (ROM) 116). Storage device 104 can also include mass storage, such as a hard disk drive.

[0024] Mass storage device 108 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), and so forth. As shown, the mass storage device in one example is a hard disk drive 124. Various drives can also be included in mass storage device 108 to enable it to read from and / or write to various computer-readable media. Mass storage device 108 includes removable media 126 and / or non-removable media. Figure 1

[0025] I / O device 110 includes various devices that enable data and / or other information to be input to or retrieved from computing device 100. Example I / O devices 110 include a cursor control device, a keyboard, a keypad, a microphone, a monitor or other display device, speakers, a printer, a network interface card, a modem, a lens, a CCD or other image capture device, and so forth.

[0026] Display device 130 includes any type of device capable of displaying information to one or more users of computing device 100. Display device 130 can be a monitor, a display terminal, or a video projection device, among others.

[0027] A graphics processing unit (GPU) 132 can be coupled to processor 102 and / or display device 130. GPU can be used to render computer-generated images and perform other graphics processing. GPU can possess some or all of the functionality of a general-purpose processor such as processor 102. GPU can also possess additional functionality specific to graphics processing. GPU can possess hard-coded and / or fixed functionality related to coordinate transformations, shading, texturing, rasterization, and other functions that aid in rendering computer-generated images.

[0028] ​The interfaces 106 include various interfaces that allow the computing device 100 to interact with various systems, devices, or computing environments. The interfaces 106 can include any number of different network interfaces 120, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include a user interface 118 and peripheral device interfaces 122. The interfaces 106 can also include one or more user interface elements 118. The interfaces 106 can also include one or more peripheral interfaces for, e.g., printers, pointing devices (mice, trackpads, etc.), keyboards, etc.

[0029] The bus 112 allows the processor 102, the memory device 104, the interfaces 106, the mass storage device 108, and the I / O device 110 to communicate with one another, or with other components or devices coupled to the bus 112. The bus 112 represents what in practice can be one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0030] In some embodiments, the processor 102 can include a cache 134, such as one or both of an LI cache and an L2 cache. Similarly, the GPU 132 can include a cache 136, which can likewise include one or both of an LI cache and an L2 cache. The processor 102 and the GPU 132 can include circuitry, or be programmed to use the caches 134, 136, to perform matrix multiplication, as described below.

[0031] To better illustrate the present application, programs and other executable program components are shown herein as discrete blocks, although such programs and components can reside at various times in different storage components of the computing device 100, and are executed by processor 102. The system and processes described herein can also be implemented in hardware, or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems or processes described herein.

[0032] Referring to Figure 2 The vector A can include one or more elements that are zero. The vector A can be a row of pixels from an image, a set of samples from an audio signal, or any other vector of values that can be affected by matrix multiplication.

[0033] As Figure 3As shown, vector A can be multiplied by matrix M. Matrix M is a two-dimensional array of values. In the example shown, matrix M has a row dimension j and a column dimension k. Each column 302 of matrix M can be multiplied by vector A, for example, through a dot product operation, where each element in vector A is multiplied by the corresponding element in column 302. All products are then summed to obtain the final result, which includes an output vector V, the number of elements of which is equal to the number of columns in matrix M.

[0034] Figure 2 and Figure 3 An example operation that can be improved based on the methods disclosed herein is shown. In some embodiments, vector A may be a set of samples from an audio signal or other sampled signal. In particular, the methods disclosed herein can be applied to any application that multiplies a vector by a matrix. The methods disclosed herein are particularly useful for very large vectors and matrices (e.g., DNA). For example, vectors with 1000 or more elements, and matrices with 1000 or more rows.

[0035] like Figure 3 As shown, one or more elements of vector A can be zero, as indicated by the shaded elements in the figure. Zero-valued elements in input vector A may be common in some applications, especially for machine learning applications (such as neural networks). The shaded row of matrix M will have no effect on matrix multiplication. In many applications, the time spent retrieving matrix M from memory is a significant source of matrix multiplication latency. Therefore, as described in the following detailed description, loading the matrix row corresponding to the zero-valued elements of vector A is omitted.

[0036] Figure 4 An example method 400 for performing matrix multiplication considering the zero-value elements of vector A is shown. Method 400 can be executed by processor 102, GPU 132, or other processing devices.

[0037] Method 400 may include step 402: loading an activation vector A from a second storage device to a first storage device. The second storage device may be RAM 114, and the first storage device may be cache 134, 136. In other embodiments, the second storage device is a mass storage device 108, and the first storage device is cache 134, 136, or RAM 114. In other embodiments, the first storage device is an L1 cache, and the second storage device is an L2 cache.

[0038] Index i can be initialized to zero in initialization step 404. Index i corresponds to the element A(i) of activation vector A and the corresponding column index j = i of matrix M to be multiplied by that element. Step 404 may include initializing a set of accumulators R(k) to zero, wherein the number of accumulators is equal to the number of columns of matrix M, such that R(k = j) corresponds to column j of matrix M.

[0039] If element A(i) is found to be non-zero in loading step 406, then row j = i of matrix M is loaded from the second storage device to the first storage device through step 408.

[0040] In some embodiments, matrix M can be compressed such that step 408 may include decompressing the rows of matrix M and storing the decompressed rows in a first storage device.

[0041] In some embodiments, matrix M can be stored as multiple entries, each entry having the form <row index><column index><element value>. This approach reduces the space required to store the sparse matrix. Therefore, loading at step 406 can include (a) loading these entries into the first storage device. For example, the location in the memory row corresponding to the column index of the row element can be replaced with data corresponding to the column index and row j. Rows of matrix M can be compressed individually so that individual rows can be decompressed and loaded. Figure 4 It is clear from the data that when A(i) = 0, the system will not attempt to load the row j = i of matrix M.

[0042] Then, method 400 may include processing step 410: processing each element M(j,k) of row j in parallel or serially, from k = 0 to kMax, where kMax is the number of columns in matrix M minus one. Specifically, for each value k, in step 412, the accumulator R(k) may be set to equal the current value of R(k) plus A(i)*M(i,k). Wherein, when processing step 410 is serial, step 412 may be omitted for values ​​of k where M(i,k) equals zero. In some embodiments, when processing step 410 is parallel, step 412 is also omitted for values ​​of k where M(i,k) equals zero. When processing step 410 is parallel, a multiply-accumulator array may be used, one for each column k, so that step 412 is performed by one of the multiply-accumulators for each value of k.

[0043] Following the processing in step 410, and if (i) is found to be zero, the method may include evaluating whether i is equal to iMax (the number of elements in A minus 1). If not, i is incremented in step 414, and processing continues in step 406. If equal, the method ends, and the values ​​in the accumulator R(k) are vectors representing the results of matrix multiplication A·M, such as...Figure 3 As shown.

[0044] refer to Figure 5 The architecture 500 and method 600 shown can be used to perform matrix multiplication, such that the multiplication of each column of C is parallelized.

[0045] like Figure 5 As shown, activation vector A and matrix M can initially be stored in second memory 501. Loader 502 loads some or all of A and M into first memory 504. Loader 502 may include zero-checker 506. Zero-checker 506 is used during the loading process (see...). Figure 4 In step 408), the value of A is evaluated. The loader 502 can store A as A' in the first memory 504, where A' does not include zero-value elements. Therefore, if A has B zero-value elements, A' will only have N B elements. If M is compressed, the loader 502 can decompress the rows of the retrieved matrix M. As with method 400 as described above, only rows in matrix M that do not correspond to zero indices of A are loaded.

[0046] The element value of each entry in a loaded row can be stored in the column position indicated by the <column> index field of that entry. In other embodiments, the entries of a row are stored in their <row index><column index><element value> format.

[0047] Sequencer 508 can manage the performance of matrix multiplication, for example, by loading elements of A' as the first argument of multiplier array 510. Sequencer 508 can load rows of matrix M as the second argument of multiplier array 510 as part of the execution step 412 for that row. Multiplier array 510 performs multiplication, and the product is then fed to accumulator 512, the accumulator for each column of matrix M, which adds the output of the multipliers to the current value of the accumulator. Sequencer 508 can instruct which multiplier outputs to sum, for example, only summing the outputs of multipliers loaded with elements of A' as the first argument, and only summing the outputs of multipliers loaded with elements of the matrix as the second argument that are not zero. The output of accumulator 512 can be written to output buffer 514 and used according to any method of matrix multiplication known in the art.

[0048] The invention may be practiced in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments should be considered illustrative in all respects only, and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All modifications falling within the meaning and equivalents of the appended claims should be included within their scope.

Claims

1. A system comprising: a multiplier array; an accumulator; a first memory; a second memory having a larger capacity and a slower processing time than the first memory; and a plurality of components, wherein the plurality of components includes a loader and a sequencer, wherein the loader is configured to: identify one or more zero index positions in an input vector such that the input vector has a value of zero at the one or more zero index positions; and load a portion of a two-dimensional array defining a first dimension and a second dimension from the second memory into the first memory, wherein the portion includes zero values and non-zero values and does not include values of the two-dimensional array having a first index in the first dimension that matches the one or more zero index positions; the sequencer is configured to: load each non-zero element of the input vector and each row of the portion of the two-dimensional array into the multiplier array in sequence to manage performance of a matrix multiplication; and cause the accumulator to accumulate only outputs of multipliers corresponding to non-zero values of each row of the portion of the two-dimensional array. the sequencer is programmed to perform a matrix multiplication of the two-dimensional array represented as M and the input vector represented as A:

2. The system of claim 1, wherein, set i = 0 and perform (a) and (b) until i equals one dimension of the two-dimensional array in the second dimension and set accumulator R(k) = 0 for k = 0 to kMax - 1, where kMax is a number of elements in the second dimension of the two-dimensional array, wherein (a) and (b) include: (a) when A(i) is not equal to zero, compute R(k) as R(k) = R(k) + A(i)*M(i,k) for k = 0 to kMax - 1; and (b) increment i by 1. the loader is further programmed to retrieve the input vector from a row of pixels obtained from an image.

3. The system of claim 1, wherein, the loader is further programmed to obtain the input vector as a set of samples from an audio signal.

4. The system of claim 1, wherein, the loader is further programmed to obtain the portion of the two-dimensional array from a plurality of entries, wherein each entry includes a first index and a second index for a non-zero value of the two-dimensional array.

5. The system of claim 1, wherein, the loader is further programmed to obtain the plurality of entries by decompressing the portion of the two-dimensional array.

6. The system of claim 5, wherein, the first memory is a cache including a processing device and located on a chip, and the second memory is a random access memory.

7. The system of claim 1, wherein, ​

Images