Enhanced multiply-accumulate device for neural networks

Abstract

A device for performing multiplication / accumulation operations has a first width and uses a computation pipeline with a second width (e.g., half the first width) to process values ​​in first and second buffers. A sequencer uses multiplication / accumulation circuitry to process combinations of partial values ​​(high-high, low-low, high-low, low-high) in the first and second buffers and adds the accumulated result of each combination to a group accumulator. The accumulated result can be left-shifted (the first width for high-high combinations and the second width for low-high and high-low combinations) before being added to the group accumulator.

Description

Technical Field

[0001] This invention relates to systems and methods for performing a large number of mathematical operations. Background Technology

[0002] One of the most common ways to improve execution speed is to execute operations in parallel, such as by using multiple processor cores. This principle can be leveraged on a larger scale by configuring graphics processing units (GPUs) with many (e.g., thousands) processing pipelines, where each pipeline can be configured to perform a mathematical function. In this way, large amounts of data can be processed in parallel. Although GPUs were originally used for graphics processing applications, they are also frequently used in other applications, especially artificial intelligence.

[0003] Improving the functionality of a GPU pipeline or any processing device that includes many processing units would be an improvement in the field. Attached Figure Description

[0004] Figure 1 This is a schematic block diagram of a computer system that can implement the methods of embodiments of the present invention.

[0005] Figure 2 This is a schematic block diagram of a multiplication / accumulation circuit according to an embodiment of the present invention.

[0006] Figure 3 This is a flowchart of a method for performing multiplication / accumulation operations on double-width input parameters according to an embodiment of the present invention.

[0007] Figure 4 This is a flowchart of another method for performing multiplication / accumulation operations on double-width input parameters, as an embodiment of the present invention.

[0008] Figure 5 This is a flowchart of a method for performing group accumulation according to an embodiment of the present invention. Detailed Implementation

[0009] To facilitate understanding of the advantages of the present invention, a more specific description of the invention will be presented with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings merely illustrate exemplary embodiments of the invention and do not constitute a limitation on its scope. The invention will now be described and explained with additional specificity and detail using the accompanying drawings.

[0010] The components of this invention can be arranged and designed in a variety of different ways. Therefore, as shown in the accompanying drawings, the following more detailed description of embodiments of the invention is not intended to limit the scope of protection claimed by the invention, but rather to illustrate the basic concept of the invention in a schematic manner. Please refer to the accompanying drawings for a better understanding of the embodiments described herein, wherein the same components are always represented by the same numbers.

[0011] Embodiments of the invention may be embodied as apparatus, method, or computer program product. Accordingly, the invention may take the form of a fully hardware embodiment, a fully software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware, which may be referred to herein as a “module” or “system.” Furthermore, the invention may be embodied in any tangible medium having computer-usable program code.

[0012] This invention can utilize any combination of one or more computer-usable or computer-readable media, including non-transitory media. For example, computer-readable media may include portable computer floppy disks, hard disks, random access memory (RAM) devices, read-only memory (ROM) devices, erasable programmable read-only memory (EPROM or flash memory) devices, portable optical disc read-only memory (CDROM), optical storage devices, and magnetic storage devices. In selected embodiments, the computer-readable medium may include any non-transitory medium that can contain, store, communicate, propagate, or transmit programs used by or in connection with an instruction execution system, apparatus, or device.

[0013] This invention can be written in any combination of one or more programming languages ​​to perform the operations of this invention. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" programming language. The program code can be executed entirely as a standalone software package on a computer system, on a standalone hardware unit, partially on a remote computer located at a distance from the computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or can establish a connection with an external computer (e.g., via the Internet through an Internet service provider).

[0014] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each part of the flowchart illustrations and block diagrams can be implemented by computer program instructions or code. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus used to produce machines, such that the instructions executed by the processor of the computer or other programmable data processing apparatus can create instructions for implementing the functions / actions specified in the flowchart illustrations and / or block diagrams.

[0015] These computer program instructions may also be stored in a non-transitory computer-readable medium that can instruct a computer or other programmable data processing device to operate in a particular manner, thereby causing the instructions stored in the computer-readable medium to produce an article of writing, which includes instruction means that can perform the functions / actions specified in the flowcharts and / or block diagrams.

[0016] Computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which are executed by the processor of the computer or other programmable data processing apparatus, provide a process for implementing the functions / actions specified in the flowchart and / or block diagram.

[0017] Figure 1 This is a block diagram of an example computing device 100. The computing device 100 can be used to perform various processes, such as those discussed herein. The computing device 100 can be used as a server, client, or any other computing entity. The computing device can perform the various functions disclosed herein and can execute one or more applications, such as those disclosed herein. The computing device 100 can be any of a variety of computing devices, such as a desktop computer, laptop computer, server computer, handheld computer, tablet computer, etc.

[0018] The computing device 100 includes one or more processors 102, one or more storage devices 104, one or more interfaces 106, one or more mass storage devices 108, one or more input / output (I / O) devices 110, and a display device 130, all coupled to a bus 112. The processor 102 includes one or more processors or controllers that execute instructions stored in the storage devices 104 and / or the mass storage devices 108. The processor 102 may also include various types of computer-readable media, such as cache memory.

[0019] Storage device 104 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 114) and / or non-volatile memory (e.g., read-only memory (ROM) 116). Storage device 104 may also include erasable and rewritable ROM, such as flash memory.

[0020] Mass storage devices 108 include various computer-readable media, such as magnetic tape, disks, optical discs, solid-state storage (e.g., flash memory), etc. Figure 1As shown, in one example, the mass storage device is a hard disk drive 124. The mass storage device 108 may also include various drives to enable it to read from and / or write to various computer-readable media. The mass storage device 108 includes removable media 126 and / or non-removable media.

[0021] I / O device 110 includes various devices that can input data and / or other information to or retrieve said data or other information from computing device 100. Example I / O devices 110 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, etc.

[0022] Display device 130 includes any type of device capable of displaying information to one or more users of computing device 100. Display device 130 may be a monitor, display terminal, or video projection device, etc.

[0023] The graphics processing unit (GPU) 132 may be coupled to the processor 102 and / or the display device 130. The GPU can be used to render computer-generated images and perform other graphics processing. The GPU may have some or all of the functions of a general-purpose processor such as the processor 102. The GPU may also have additional graphics processing-specific functions. The GPU may have hard-coded and / or fixed-line graphics functions related to coordinate transformation, shading, texturing, rasterization, and other functions that help render computer-generated images.

[0024] Interface 106 includes various interfaces that allow computing device 100 to interact with other systems, devices, or computing environments. Interface 106 may include any number of different network interfaces 120, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include user interfaces 118 and peripheral device interfaces 122. Interface 106 may also include one or more user interface elements 118. Interface 106 may also include one or more peripheral interfaces for, for example, printers, pointing devices (mice, touchpads, etc.), keyboards, etc.

[0025] Bus 112 allows processor 102, memory device 104, interface 106, mass storage device 108, and I / O device 110 to communicate with each other or with other components or devices coupled to bus 112. Bus 112 represents one or more of several types of bus architectures, such as system bus, PCI bus, IEEE 1394 bus, USB bus, etc.

[0026] In some embodiments, processor 102 may include cache 134, such as one or both of L1 and L2 caches. Similarly, GPU 132 may include cache 136, which may also include one or both of L1 and L2 caches.

[0027] For the purpose of better illustrating the invention, programs and other executable program components are shown herein as discrete blocks, although such programs and components may reside in different storage components of computing device 100 at different times and be executed by processor 102. The systems and processes described herein may also be implemented in hardware, or a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) may be programmed to perform one or more of the systems or processes described herein.

[0028] Reference Figure 2 In some embodiments, GPU 132, processor 102, or other computing devices may include access buffers 200, 202, such as those included in caches 134, 316, RAM 114, or some other hardware location. Values ​​stored in buffers 200, 202 have a first width, such as 16 bits. Other portions of the computing pipeline of GPU 132, processor 102, or other computing devices may have a smaller width, such as half the first width, such as 8 bits, where the first width is 16 bits. Values ​​stored in buffers 200, 202 may be values ​​used to implement and apply convolutional neural networks (CNNs) or other types of neural networks. For example, buffer 200 may store the coefficients of a CNN, while buffer 202 may store the activation values ​​of a CNN (e.g., values ​​processed according to the CNN). The manner in which CNN processing is performed may be any method known in the art, wherein some or all of the multiplication / accumulation operations are performed according to the methods of this disclosure.

[0029] Sequencer 204 can read values ​​from buffers 200 and 202 to perform multiplication / accumulation operations using the values ​​stored in buffers 200 and 202. Specifically, sequencer 204 can output a series of parameters 206 and 208, which have a second width and are a portion of a first value from buffer 200 and a second value from buffer 202. See below. Figures 3 to 5 Describe in detail how the sequencer generates parameters 206 and 208.

[0030] Parameters 206 and 208 are input into computation pipeline 210, which is configured to perform multiplication / accumulation operations. For this purpose, pipeline 210 may include a multiplier 212 that multiplies parameters 206 and 208 to produce a product, and an adder 214 that adds the product to the contents of an accumulation buffer 216 to obtain a sum and writes the sum to the accumulation buffer 216.

[0031] As described below, the contents of the accumulator buffer 216 can be added to the contents of the group accumulator buffer 220 using adder 218, and the result of the addition can be written to the group accumulator buffer 220. This operation will be described in detail below. The group accumulator buffer 220 can be much wider than the accumulator buffer 216. For example, when the first width is 16 bits and the second width is 8 bits, the group accumulator buffer can have a width of 48 bits, while the accumulator buffer 216 has a width of 24 bits.

[0032] The content of the group cumulative variable 220 is, according to Figures 3 to 5 The method involves multiplying and summing the values ​​from registers 200 and 202. The contents of the accumulated variable 220 can then be used for any desired purpose, such as implementing a CNN or any other process that might benefit from the multiplication / accumulation operation. Specifically, in any scenario where these operations are performed, the multiplication / accumulation operation can be used to implement dot products or matrix multiplication.

[0033] Reference Figure 3 The bit positions of each value stored in buffers 200 and 202 can define a high part and a low part. The high part has a higher magnitude (e.g., higher importance) than the low part and does not overlap with the low part. The number of bits in the high part and the number of bits in the low part are equal to the number of bits of each value stored in buffers 200 and 202. For example, in the case of storing 16-bit values ​​in buffers 200 and 202, bit positions 8 to 15 may be high bits, and bit positions 0 to 7 may be low bits, where bit position 0 is defined as the least significant bit (LSB).

[0034] Method 300 is one method for implementing multiplication / accumulation operations. For the purposes of the following discussion, AHi represents the high portion at buffer position i of buffer 200. BHi represents the high portion at buffer position i of buffer 202. ALi represents the low portion at buffer position i of buffer 200. BLi represents the low portion at buffer position i of buffer 202.

[0035] Method 300 may include performing a multiplication / accumulation operation (302) on AHi and BHi for all buffer positions i. Specifically, sequencer 204 may output AH according to pipeline 210 for each value i from 0 to N-1 (where N is the number of values ​​to be processed).i and BH i As parameters 206 and 208 to be processed, the result stored in the accumulator buffer 216 after step 302 will be...

[0036] Then, method 300 may include adding the contents of accumulator buffer 216 to the contents of group accumulator buffer 220 (304) and writing the result of the addition to group accumulator buffer 220. Before performing method 300, group accumulator buffer 220 and accumulator buffer 216 may be initialized to zero, such that step 304 includes simply writing the contents of accumulator buffer 216 to group accumulator buffer 220. As follows regarding Figure 5 The description suggests that writing may include shifting the contents of the accumulator buffer by a first width (e.g., 16 bits) before addition to account for the fact that high-resolution AHi and BHi have already been processed.

[0037] Method 300 may include performing a multiplication / accumulation operation (306) on AHi and BLi for all buffer positions i. Specifically, sequencer 204 may output AHi and BLi as parameters 206, 208 to be processed for each value i from 0 to N-1 (where N is the number of values ​​to be processed) according to pipeline 210. Therefore, the result stored in the accumulation buffer 216 after step 306 will be

[0038] Then, method 300 may include adding the contents of the accumulator buffer 216 to the contents of the group accumulator buffer 220 (step 308), and writing the sum to the group accumulator buffer 220. Before performing method step 306, the accumulator buffer 216 may be initialized to zero. As follows regarding... Figure 5 As described, the addition step 308 may include shifting the contents of the accumulation buffer 216 by a second width (e.g., 8 bits) before addition to account for the fact that the high part AHi has been processed.

[0039] Method 300 may include performing a multiplication / accumulation operation of 310ALi and BLi for all buffer locations i (step 310). Specifically, sequencer 204 may output ALi and BLi as parameters 206, 208 to be processed for each value i from 0 to N-1 (where N is the number of values ​​to be processed), according to pipeline 210. Therefore, the result stored in accumulation buffer 216 after step 310 will be

[0040] Then, method 300 may include adding the contents of the accumulator buffer 216 to the contents of the group accumulator buffer 220 (step 312), and writing the sum to the group accumulator buffer 220. Before performing method step 310, the accumulator buffer 216 may be initialized to zero. As follows regarding... Figure 5 The addition step 312 will not include the contents of the shift-accumulator buffer 216, because only the low-precision portion ALi, BLi is processed.

[0041] Method 300 may include performing a multiplication / accumulation operation on ALi and BHi for all buffer positions i (step 314). Specifically, sequencer 204 may output ALi and BHi as parameters 206, 208 for each value i from 0 to N-1 (where N is the number of values ​​to be processed) to be processed according to pipeline 210. Therefore, the result stored in accumulation buffer 216 after step 306 will be

[0042] Then, method 300 may include: adding the contents of accumulator buffer 216 to the contents of group accumulator buffer 220 (step 316), and writing the sum to group accumulator buffer 220. Accumulator buffer 216 may be initialized to zero. As follows regarding... Figure 5 As described, the addition step 316 may include shifting the contents of the accumulation buffer 216 by a second width (e.g., 8 bits) before addition to account for the fact that the high part BHi has been processed.

[0043] After executing method 300, the results of multiplication / accumulation operations performed on all values ​​at buffer positions 0 to N-1 in buffers 200 and 202 are stored in buffers 200 and 202 by the group accumulation buffer 220. Note that the order of steps 302, 306, 310, and 314 is arbitrary and can be rearranged and interchanged. Also note that while buffer positions 0 to N-1 were mentioned above, the starting address of this method and the other methods described herein can be any location in the memory defining the buffer.

[0044] Figure 4 A more detailed method 400 is shown, which uses a computation pipeline 210 with a second width that is half the first width to perform multiplication / accumulation operations on values ​​with the first width.

[0045] Method 400 may include setting the position of the first parameter 206 low, that is, the low portion of the first value in buffer 200 (step 402). Method 400 may further include setting the position of the second parameter 208 low (step 404), that is, the low portion of the second value in buffer 202. In this example, the low portion is processed first; however, this is merely exemplary and it is also possible to start from the high portion.

[0046] Method 400 may include initializing the current buffer position to zero and initializing the accumulator buffer 216 to zero (step 406).

[0047] Then, method 400 may include performing a multiplication / accumulation operation on a portion of the first value at the first parameter position and a portion of the second value at the second parameter position (step 408). For example, step 408 may include using computation pipeline 210 to perform the following operations:

[0048]

[0049] in

[0050] If the first parameter is in a high position, then P1 = 1; if the first parameter is in a low position, then P1 = 0.

[0051] If the first parameter is in a high position, then P2 = 1; if the first parameter is in a low position, then P2 = 0.

[0052] S = Second width.

[0053] The calculation in step 408 can be performed iteratively, for example, starting from i = 0, (a) performing multiplication A i [S(P1+1)-1:S*P1]*B i [S(P2+1)-1:S*P2] to obtain the product, add the product to the accumulator buffer 216 to obtain a sum, and write the sum to the accumulator buffer 216, and (b) if i is not equal to N-1, increment i and repeat (a).

[0054] Then, method 400 may include adding the contents of accumulator buffer 216 to the contents of group accumulator buffer 220 (step 410) and writing the sum to group accumulator buffer 220. Method 400 may be performed before initializing group accumulator buffer 220.

[0055] Then, method 400 may include evaluating whether the second parameter position is high (step 414). If not, the second parameter position 412 is set high, and the process continues from step 406. If yes, method 400 may further include evaluating whether the first parameter position is high (step 416). If not, the first parameter position is set high (step 418), and the process continues from step 404. If yes, method 400 ends, and the value stored in the group accumulation buffer 220 is the result of multiplying / accumulating the values ​​from 0 to N-1 in buffers 200 and 201. As mentioned above, positions 0 to N-1 are merely exemplary, and any range of memory addresses can be processed according to this method. It should also be noted that the address range of buffer 200 may be the same as or different from the address range in buffer 202 processed according to method 400.

[0056] Reference Figure 5 When adding the contents of accumulator buffer 216 to the contents of group accumulator buffer 220, the illustrated method 500 can be used. Method 500 may include evaluating whether both the first and second parameter positions are high at step 502. If so, the contents of accumulator buffer 216 are shifted left (assuming the leftmost bit is the most significant) by a first width (step 504), for example, 16 bits, and then the shifted value at step 504 is added to the contents of group accumulator buffer 220 (step 506), and the result of step 506 is then written to group accumulator buffer 220.

[0057] Method 500 may include evaluating whether only one of the first and second parameter positions is high (step 508). If so, the contents of the accumulator buffer 216 are shifted left (assuming the leftmost bit is the most significant) by a second width (step 510), for example, 8 bits, and then the shifted value at step 510 is added to the contents of the group accumulator buffer 220 (step 506), and the result of step 506 is then added to the group accumulator buffer 220.

[0058] If neither parameter is high, no shift is performed, and step 506 is executed on the unshifted contents of the accumulator buffer 216.

[0059] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An apparatus comprising: The first input buffer is configured to store a first value having a first width; The second input buffer is configured to store a second value having a first width; The multiplication / accumulation circuit is configured to perform a multiplication / accumulation operation on an input parameter having a second width, which is half the first width; The group accumulator is configured to accumulate output multiplier / accumulator circuitry. and A sequencer, wherein for each of a plurality of combinations of possible partial positions of the first and second input buffers, the sequencer is used to: The corresponding portions of the first and second values ​​are input into the multiplication / accumulation circuit; and The group accumulator is invoked to add the output of the multiplication / accumulation circuit to the contents of a group accumulator buffer; The partial locations of the first input buffer and the second input buffer include: A high part and a low part, wherein the high part has a higher amplitude than the low part, the high part and the low part do not overlap, and the sum of the number of bits in the high part and the number of bits in the low part is equal to the number of bits in the first width; Among them, the possible combinations of partial positions of the first input buffer and the second input buffer include: The high part of the first value in the first input buffer and the high part of the second value in the second input buffer; The lower part of the first value in the first input buffer and the higher part of the second value in the second input buffer; The lower part of the first value in the first input buffer and the lower part of the second value in the second input buffer; The high part of the first value in the first input buffer and the low part of the second value in the second input buffer; The group accumulator is further configured as follows: When each combination of partial positions of the first input buffer and the second input buffer includes the high portion of the first input buffer and the high portion of the second input buffer, the output of the multiplication / accumulation circuit is shifted by a first width to obtain the shifted output, and the shifted output is added to the contents of the group accumulator buffer; When each combination of partial positions of the first input buffer and the second input buffer includes only the high portion of one of the first input buffer and the second input buffer, the output of the multiplication / accumulation circuit is shifted by a second width to obtain the shifted output, and the shifted output is added to the contents of the group accumulator buffer.

2. The device according to claim 1, wherein, The first width is 16 bits, and the second width is 8 bits.

3. The device according to claim 2, wherein, The group accumulation buffer has a width of 48 bits.

4. The device of claim 1, further comprising a controller programmed to implement a convolutional neural network using the first input buffer, multiplication / accumulation circuitry, sequencer, and group accumulator.

5. The device of claim 1, further comprising a graphics processing unit, the graphics processing unit including the first input buffer, a multiplication / accumulation circuit, a sequencer, and a group accumulator.

6. The device according to claim 1, wherein, The multiplication / accumulation circuit is a first multiplication / accumulation circuit, and the device further includes: a plurality of multiplication / accumulation circuits comprising the first multiplication / accumulation circuit.

7. A method comprising: A first input buffer is provided, which is configured to store a first value having a first width; A second input buffer is provided, which is configured to store a second value having a first width; A computation pipeline with a second width is used to perform multiplication / accumulation operations on the high parts of the first value and the high parts of the second value to obtain a first intermediate accumulation; Based on the first intermediate accumulation, increase the group accumulation value stored in the group accumulation buffer; Perform multiplication / accumulation operations on the high part of the first value and the low part of the second value to obtain the second intermediate accumulation; Based on the second intermediate accumulation, increase the group accumulation value stored in the group accumulation buffer; Perform multiplication / accumulation operations on the lower parts of the first and second values ​​to obtain the third intermediate accumulation; Based on the third intermediate accumulation, increase the group accumulation value stored in the group accumulation buffer; Perform multiplication / accumulation operations on the lower part of the first value and the upper part of the second value to obtain the fourth intermediate accumulation; as well as Based on the fourth intermediate accumulation, increase the group accumulation value stored in the group accumulation buffer; Wherein, the higher part of the first value and the second value has a higher amplitude than the lower part, the higher part and the lower part do not overlap, and the sum of the number of digits in the higher part and the number of digits in the lower part is equal to the number of digits in the first width; The incrementing of the group accumulation value stored in the group accumulation buffer based on the first intermediate accumulation includes: The group accumulator value stored in the group accumulator buffer is incremented by the first intermediate accumulator with the first width shifted to the left.

8. The method according to claim 7, wherein, The second width is half of the first width.

9. The method according to claim 8, wherein, The first width is 16 bits, and the second width is 8 bits.

10. The method according to claim 9, wherein, The group accumulation buffer has a width of 48 bits.

11. The method according to claim 7, characterized in that, Increasing the group accumulation value stored in the group accumulation buffer based on the second intermediate accumulation includes: The group accumulation value stored in the group accumulation buffer is increased by the second intermediate accumulation with the second width shifted to the left.

12. The method according to claim 11, characterized in that, Increasing the group accumulation value stored in the group accumulation buffer based on the third intermediate accumulation includes: The group accumulator value stored in the group accumulator buffer is incremented by the third intermediate accumulator without first shifting the third intermediate accumulator to the left.

13. The method of claim 7, further comprising using the group accumulated values ​​to implement a convolutional neural network.

14. The method according to claim 13, wherein, The first value is a coefficient of the convolutional neural network, and the second value is an activation value processed by the convolutional neural network.

Images