An efficient multiply-add unit for convolutional neural network processing including max pooling

Abstract

A system and method for an enhanced multiply-accumulate unit for convolutional neural network processing. An exemplary multiply-accumulate unit includes a multiplier configured to receive (1) an input value for a window of input values, and (2) a weight value set to a negative value. The unit further includes an adder configured to add a result received from the multiplier with a value in an accumulator of a MAC unit. The unit further includes a multiplexer configured to select to output (1) a result of the adder and (2) the input value, and an accumulator configured to receive an output from the multiplexer, and configured to be enabled or disabled according to a result of the adder based on which MAC unit is performing a particular convolution operation.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 316,804, entitled "EFFICIENT MULTIPLY-ACCUMULATE UNITS FOR CONVOLUTIONAL NEURAL NETWORK PROCESSING INCLUDING MAX POOLING," filed March 4, 2022, the disclosure of which is incorporated by reference herein in its entirety.

[0002] The present disclosure relates to hardware processors, and more particularly to extended multiply-accumulate units. [Background technology]

[0003] Neural networks rely on entirely different applications and increasingly form the basis of technology. For example, neural networks can be utilized to perform object classification on images acquired via a user device (e.g., a smartphone). In this example, the neural network can represent a convolutional neural network that applies convolutional layers, pooling layers, and one or more fully connected layers to classify objects depicted in the image. As another example, neural networks can be utilized for translation of text between languages. In this example, the neural network can represent a recurrent neural network, a transformer network, etc.

[0004] Typically, a graphics processing unit (GPU) can be used during training or inference. As can be appreciated, a GPU can perform the kind of quick mathematical calculations that are suited to a particular layer of a neural network. For example, a GPU can quickly calculate a forward pass through a fully connected layer based on multiplying different vectors. However, GPUs are not optimized for a particular neural network use case. This can reduce the complexity that allows a neural network to achieve an inference time that is less than a particular time constraint. For example, a neural network that performs object recognition on images acquired from a camera may be limited in the number of images it can process per second. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary prior art matrix processor configured to perform convolutional neural network operations.

[0006] [Figure 2A] FIG. 1 is a block diagram illustrating an example matrix processor including an enhanced multiply-accumulate unit (MAC unit) in accordance with the techniques described herein.

[0007] [Figure 2B] FIG. 2 is a block diagram illustrating details of an exemplary MAC unit.

[0008] [Diagram 3] 1 is a flowchart of an example process for performing convolutional neural network operations such as max pooling using an enhanced MAC unit as described herein.

[0009] [Figure 4] 1 is a block diagram illustrating an example vehicle including a vehicle processor system.

[0010] The embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description: Like reference numerals have been used to identify like elements shown in one or more of the drawings, and it should be understood that the indications therein are for the purpose of illustrating the embodiments of the present disclosure and are not intended to limit the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present application describes an extended multiply-accumulate (MAC) unit that may be included in a matrix processor, optionally with other MAC units. In some embodiments, the matrix processor may perform convolutional neural network processing (e.g., the matrix processor may be a convolution engine). For example, the matrix processor may calculate a forward pass of information through layers forming a convolutional neural network. Exemplary information may include images from an image sensor, such as an image sensor located around an autonomous or semi-autonomous vehicle. As described below, the extended MAC unit described herein may enable the performance of certain operations commonly used in processing associated with convolutional neural networks. Such operations may include, for example, pooling, such as max pooling.

[0012] An exemplary hardware processor, such as a convolution engine, can be configured for efficient processing of a convolutional neural network. For example, the hardware processor can receive image data (e.g., vectorized data) and a filter or kernel to be applied to the image data. In this example, the hardware processor can have a matrix processor having a matrix of MAC units configured to calculate at least a dot product between the image data and the filter or kernel. An output of the matrix processor can represent an output of a convolutional layer included in the convolutional neural network. The output can also represent an output associated with an individual output channel. A nonlinear activation function can then be applied to the aforementioned output, for example, via a different hardware unit or element.

[0013] With respect to pooling, the hardware processor may include another hardware unit or element that performs the pooling. For example, with respect to a max pooling layer, the hardware unit or element may identify the maximum value within a window of values ​​included in the input to the max pooling layer. An exemplary window may be, for example, 2×2 with a stride of 2, such that the max pooling layer subsamples all depth slices in the input by 2 along both width and height. As an example, the max pooling layer identifies the maximum value of a first 2×2 input portion, and then identifies the maximum value of a second 2×2 input portion that is two widths away from the first portion.

[0014] As can be appreciated, the use of a separate hardware unit or element to perform post-processing such as max pooling can increase the complexity and die space utilized by the hardware processor. Additionally, for electric vehicles, power requirements may be greater.

[0015] Described herein is an exemplary enhanced MAC unit that enables improved performance associated with performing max pooling. As described below, the enhanced MAC unit may include exemplary hardware that enables performance of max pooling. Such hardware may include, for example, a multiplexer, such as a 2-to-1 multiplexer, that multiplexes (1) input data and (2) the output of an addition operation that adds an accumulation value to a multiplication of the input data and weight data (e.g., kernel or filter data). The multiplexer may select between two multiplexing sources depending on whether a first operation (e.g., convolution) or a second operation (e.g., max pooling) is being performed.

[0016] To enable max pooling, and thus comparison of values ​​within a window of input values, the enhanced MAC unit may receive (1) an input value within a window of input values, and (2) a weight of negative one. Thus, a multiplication of these values ​​may represent the negation of the input value. This multiplication may be added to an accumulated value, such as a previous input value within the window of input values, and a comparison may be identified based on the addition. For example, a high bit of the result may indicate whether the input value is greater than the previous value, or whether the input value is less than the previous value. If the result is greater, the input value may be stored as an accumulated value (e.g., an accumulator element). For example, an accumulator may be enabled based on the value of the high bit. If the result is less, the accumulated value may remain at the previous value. For example, an accumulator may be disabled based on the value of the high bit. Thus, in these examples, the high bit may be input to an enable / disable of the accumulator.

[0017] Thus, the enhanced MAC units described above can be utilized to reduce the complexity and / or die size associated with matrix processors used for autonomous or semi-autonomous driving. Indeed, coordination of the MAC units through the use of multiplexers may allow for more efficient processing of information and reduced complexity.

[0018] The MAC unit may be included in a matrix processor, such as those described in U.S. Pat. No. 11,157,287, U.S. Patent Application Publication No. 2019 / 0026250, and U.S. Pat. No. 11,157,441, which are incorporated by reference in their entireties and form part of this disclosure as if set forth herein. In some embodiments, the matrix processor is a non-systolic array of MAC units, where the input data is provided to one direction of the matrix processor and the weight data is provided to a second direction of the matrix processor.

[0019] In some embodiments, the MAC units (e.g., enhanced MAC units) described herein may be arranged such that a tree-like reduction may be performed. For example, a first number of MAC units may determine the maximum value of a respective portion of an input window. As an example, four MAC units may determine the maximum value of a 4x4 sub-matrix of an 8x8 input window. The output of these four MAC units (e.g., four values) may be provided to a single MAC unit to determine the maximum value. It is also possible to provide the output of two MAC units (e.g., two values ​​provided to each MAC unit on successive clock cycles, etc.) followed by one MAC unit to determine the maximum value.

[0020] As can be appreciated, numerous topologies of the MAC units described herein may be used in a processor and fall within the scope of the disclosure herein.

[0021] In some embodiments, techniques described herein can be used to determine the minimum value within the pooling window. For example, the accumulator can initially store a default value (e.g., the maximum representable value) as described herein. In this example, the input data value multiplied by a negative value can be added to the default value, and the accumulator can be allowed to store the input data value based on the high bit indicating that the addition is positive.

[0022] Block Diagram 1 is a block diagram illustrating an exemplary prior art matrix processor 100 configured to perform convolutional neural network operations. The exemplary matrix processor 100 includes a number of multiply-accumulate units (MAC units) capable of processing the convolutional layers of a convolutional neural network.

[0023] As an example, the matrix processor 100 may receive input data along a direction A 102 of the matrix processor 100. For example, the input data may represent a vectorized form of one or more images acquired from an image sensor. As another example, the matrix processor 100 may receive weight data (e.g., one or more filters, kernels) along a direction B 104 of the matrix processor 100. For example, the weight data may represent a vectorized form of the weight data.

[0024] The matrix processor 100 may use multiple MAC units to compute one or more convolutions associated with the input data and the weight data. For example, the MAC units may compute dot products of portions of the input data and the weight data. These dot products may be used to generate one or more convolutions.

[0025] An exemplary MAC unit 110 is included in FIG. 1. As shown, the MAC unit 110 includes a first element that multiplies input data with weight data. The MAC unit 110 also includes a second element that adds the result of the first element to an accumulated value stored in an accumulator. In this manner, multiplications of the weight data with the input data can be added over clock cycles of the matrix processor 100.

[0026] Although matrix processor 100 can efficiently perform convolutions using a MAC unit, the MAC unit lacks the ability to perform other operations commonly relied upon in convolutional neural networks. For example, matrix processor 100 may require additional hardware elements to perform max pooling.

[0027] 2A is a block diagram illustrating an example matrix processor 200 including an enhanced multiply-accumulate unit (MAC unit) in accordance with the techniques described herein. The matrix processor 200 includes an example of an enhanced MAC unit 210 in accordance with the techniques described herein. The example matrix processor 200 may be used to perform convolutional neural network processing, for example, with one output channel active for a given input channel of data. As can be appreciated, this can accommodate conventional operations (e.g., pooling, which may be conventionally depth-wise).

[0028] The extended MAC unit 210 may be used to calculate or determine a maximum value for a set of input values. In the illustrated embodiment, the extended MAC unit 210 receives input values ​​212. For example, the input values ​​212 may represent input values ​​from a window of input values ​​associated with the size of the max pooling operation being performed. Exemplary sizes may include 2×2 operations, 3×3 operations, etc. Values ​​within the window of input values ​​may be provided consecutively to the extended MAC unit 210 to enable comparison of these input values.

[0029] The enhanced MAC unit 210 further receives a negative weight value, thus negating the input value 212. For example, the matrix processor 200, or a controller associated with the processor 200, may negate the weight. In this example, this weight value may be entered by an instruction associated with performing max pooling. As described below, with respect to FIG. 2B, the negative input value 212 may be added to a previous input value in an accumulator. Thus, the result of this addition may indicate whether the input value 212 is greater than the previous input value.

[0030] The enhanced MAC unit 210 further includes a multiplexer 216. During processing of a convolutional layer, the multiplexer 216 may be configured to select from a first source (e.g., the output of the summation). For example, the matrix processor 200 may execute a software or hardware command or instruction that causes the selection of the first source. In this manner, the multiplexer 216 may pass the value output from the summation to an accumulator for inclusion or storage. During processing of a max pooling layer, the multiplexer 216 may instead be configured to select from a second source (e.g., the input data). Thus, the input data may be stored in the accumulator depending on whether the input data is greater (e.g., greater, greater than) the previous input value.

[0031] Figure 2B is a block diagram illustrating details of an exemplary enhanced MAC unit 210. As discussed above, with respect to Figure 2A, input data 212 (e.g., input values) may be provided to the enhanced MAC unit 210 with weight data 214 set to negative values. For example, a software or hardware operation may cause the weight data 214 to be set to negative values ​​when the matrix processor 200 is performing a max pooling operation.

[0032] In the illustrated example, the multiplexer 216 is configured to select a particular source (e.g., input data 212) via a selector 218. For example, the selector 218 may switch between sources based on whether the matrix processor 200 is processing a convolutional layer or a max pooling layer. In this example, the selector 218 may be set based on the instruction being executed.

[0033] An exemplary operation of the Enhanced MAC unit 210 may include input data 212, which is the first value in a window of input values. In this example, there may be no accumulated value so that the value is set to a default (e.g., 0). When the input data 212 is added to the default, it is multiplied with a negative value, resulting in the output of the multiplication. This first value may be positive in some embodiments (e.g., assuming the input is image data), so the result of the addition will be negative. For example, if the first value is 7, the multiplication will be negative 7 and the result of the addition will remain negative 7.

[0034] With respect to default values, the accumulator may be reset before determining the maximum value within a window of values. For example, when performing the maximum of a series of unsigned values, the accumulator may be reset to 0 (e.g., the smallest unsigned value). As another example, when performing the maximum signed value, the reset value may be the smallest negative number that can be represented by the MAC's input data width (e.g., 0xffff...). An alternative to these reset cases is to simply override the circuitry to bring the first data into the accumulator, so no reset value is needed.

[0035] In the above example, the result of the addition is negative, so the high bit 220 is set to a particular value (e.g., a value related to a signed magnitude representation). This high bit may be used to enable or disable an accumulator, for example, when processing a max pooling layer. As an example, the high bit 220 is 1, so the accumulator is enabled. Thus, the input data 212 is forwarded through the multiplexer 216 and stored or included in the accumulator.

[0036] The next input value within the window of input values ​​can then be obtained. In some embodiments, the next input value may be provided in a subsequent clock cycle. As above, the next input value is multiplied by a negative value and the result of the multiplication is added to the accumulated value. If the next input value is greater than the accumulated value, then the high bit 220 is set to 1 and the accumulator stores the next input value. However, if the next input value is less than the accumulated value, then the high bit 220 is set to 0 and the accumulator is disabled. Thus, the accumulated value remains.

[0037] When the final value within the aforementioned window of values ​​is processed through the Extended MAC unit 210, the value stored in the accumulator represents the maximum value within the window of values.

[0038] In some embodiments, when processing convolutional layers, the high bit 216 may be ignored or set to 1. In this way, the accumulators can function normally. However, when processing pooling layers, the high bit may be used to indicate whether the accumulator is enabled or disabled.

[0039] Exemplary Flowchart 3 is a flow chart of an example process 300 for performing convolutional neural network operations such as max pooling using the enhanced MAC units described herein. For convenience, the process 300 is described as being performed by a matrix processor that includes multiple enhanced multiply-accumulate units (MAC units).

[0040] At block 302, the matrix processor causes the convolution engine to perform max pooling. The matrix processor, sometimes referred to as the convolution engine, may execute one or more instructions associated with performing max pooling. For example, in some embodiments, the max pooling may be defined via instructions such as input data, window size, stride, etc. The input data may represent output from a previous layer of the neural network, which may be a previous pooling layer, a previous convolution layer, etc.

[0041] As described above, the extended MAC unit may be configured for use with max pooling. For example, a selector (e.g., selector 218) of a multiplexer included in the extended MAC unit may be set to a particular value that causes the selection of input data. In this example, the output of the multiplexer may be routed to an accumulator within the extended MAC unit. Thus, the extended MAC unit may be configured to store the input data routed through the multiplexer.

[0042] At block 304, the matrix processor provides a portion of the input data to the extended MAC unit. As discussed above, max pooling can refer to identifying respective maximum values ​​within multiple windows of input values. For example, the size of the max pooling window may be 2×2 and the stride may be set to 2. In this example, for ease of reference, the input data may include four windows (e.g., the input data may have 16 values). In some embodiments, each of the extended MAC units may operate on a respective window. With respect to the four windows, the four extended MAC units may identify respective maximum values ​​within the four windows.

[0043] The matrix processor causes a comparison of the input data in block 306. As described in Figures 2A-2B, each enhanced MAC unit may identify the maximum value within a respective window.

[0044] Vehicle block diagram 4 illustrates a block diagram of a vehicle 400 (e.g., vehicle 102). The vehicle 400 may include one or more electric motors 402 that cause the vehicle 400 to move. The electric motors 402 may include, for example, induction motors, permanent magnet motors, etc. As known to those skilled in the art, batteries 404 (e.g., one or more battery packs each including multiple batteries) may be used to power the electric motors 402.

[0045] The vehicle 400 further includes a propulsion system 406 that can be used to set the gear (e.g., propulsion direction) of the vehicle. For an electric vehicle, the propulsion system 406 can regulate the operation of the electric motor 402 to change the propulsion direction.

[0046] Additionally, the vehicle includes a matrix processor 200 that includes a number of extended multiply-accumulate units (MAC units) as described herein. The matrix processor 200 can process data such as images received from image sensors (e.g., cameras 104A-104N) positioned around the vehicle 400. The vehicle processor system 100 can additionally output information to, and receive information (e.g., user input) from, a display 408 included in the vehicle 400.

[0047] Other embodiments All of the processes described herein may be embodied and fully automated via software code modules executed by a computing system including one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

[0048] Many other variations beyond those described herein will be apparent from the present disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different order, or may be added, merged, or omitted entirely (e.g., not all acts or events described may be necessary to the implementation of an algorithm). Furthermore, in certain embodiments, operations or events may be performed simultaneously rather than sequentially, for example, via multi-threaded processing, interrupt processing, or multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may be performed by different machines and / or computing systems that can function together.

[0049] The various exemplary logic blocks, modules and engines described in connection with the embodiments disclosed herein may be implemented or executed by machines such as processing units, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, but in alternative examples, the processor may be a controller, microcontroller, or state machine, combinations thereof, and the like. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, the processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuitry or mixed analog and digital circuitry. The computing environment may include any type of computer system, including, but not limited to, a computer system based on a computational engine within a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or appliance, to name a few.

[0050] In particular, conditional language such as "can, could," "might, may," and the like, unless otherwise specified, is understood within the context in which it is generally used to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is generally not intended to imply that the features, elements, and / or steps are somehow required by one or more embodiments, or that one or more embodiments necessarily include logic for determining whether or not those features, elements, and / or steps should be included in or performed in any particular embodiment, with or without user input or prompting.

[0051] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is understood within the context in which it is generally used to state that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y and / or Z), unless otherwise indicated. Thus, such disjunctive language is generally not intended to, and should not, imply that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z, respectively, to be present.

[0052] Any process descriptions, elements or blocks in the flow diagrams described herein and / or shown in the accompanying drawings should be understood as potentially representing modules, segments or portions of code that include one or more executable instructions for implementing a particular logical function or element in the process. As will be appreciated by those skilled in the art, alternative implementations in which elements or functions may be omitted, performed or described in a different order than that shown or described, including substantially simultaneously or in reverse order, depending on the functionality involved, are included within the scope of the embodiments described herein.

[0053] Unless otherwise noted, articles such as "a" or "an" should generally be construed to include one or more of the listed items. Thus, a phrase such as "a device configured to" is intended to include the enumerated device or devices. Such one or more enumerated devices may also be collectively configured to perform the stated enumeration. For example, "a processor configured to perform enumerations A, B, and C" may include a first processor configured to perform enumeration A working in conjunction with a second processor configured to perform enumerations B and C.

[0054] It should be emphasized that many variations and modifications can be made to the above-described embodiments, and that the elements are to be understood as being among the other acceptable examples, and all such modifications and variations are intended to be included within the scope of the present disclosure.

Claims

1. a multiply-accumulate unit (MAC unit) included in a matrix processor configured to compute a forward pass through a neural network, the matrix processor configured to be included in a vehicle, the MAC unit comprising: a multiplier configured to receive (1) an input value for a window of input values, and (2) a weight value; an adder configured to add a result received from the multiplier with a value in an accumulator of the MAC unit; a multiplexer configured to select to output (1) the result of the adder and (2) the input value; the accumulator, configured to receive and store an output from the multiplexer when the accumulator is enabled; Based on the MAC unit performing a max pooling operation, the weight value is set to negative one; the multiplexer is configured to select the input value to output; and an accumulator, the accumulator configured to be enabled or disabled based on a result of the adder.

2. A MAC unit as described in claim 1, wherein the MAC unit is configured to successively receive individual input values ​​within the window of input values ​​based on the MAC unit performing a max pooling operation.

3. The MAC unit of claim 2 , wherein the MAC unit is configured to identify a maximum value within the window of input values.

4. A MAC unit as described in claim 1, wherein the multiplexer is configured to select the result of the adder based on the MAC unit performing processing associated with a convolutional layer.

5. A MAC unit as described in claim 1, wherein the accumulator is set to be valid during execution of a process associated with a convolutional layer based on the MAC unit performing the process.

6. The MAC unit of claim 1, wherein a high bit of the result of the adder is used to enable or disable the accumulator.

7. The MAC unit of claim 1 , wherein the accumulator is configured to store the input value based on the input value being greater than a different value stored in the accumulator.

8. The MAC unit of claim 7 , wherein the accumulator is configured to be enabled such that the accumulator is configured to store the input value.

9. The MAC unit of claim 1 , configured to be disabled such that the accumulator does not store the input value based on the input value being less than a different value stored in the accumulator.

10. A matrix processor comprising a plurality of multiply-accumulate units (MAC units) according to claim 1.

11. 11. A matrix processor as claimed in claim 10, wherein each MAC unit is configured to identify the maximum value of different windows of input values ​​of a plurality of windows of input values.

12. 1. A method implemented by a matrix processor, comprising: obtaining information indicating that the matrix processor performs max pooling; providing a portion of the input data to a respective multiply-accumulate unit (MAC unit), each MAC unit comprising: a multiplier configured to receive (1) an input value of a respective portion of the input value, and (2) a weight value; an adder configured to add a result received from the multiplier with a value in an accumulator of the MAC unit; a multiplexer configured to select to output (1) the result of the adder and (2) the input value; the accumulator, configured to receive and store an output from the multiplexer when the accumulator is enabled; Based on the MAC unit performing a max pooling operation, the weight value is set to negative one; the multiplexer is configured to select the input value to output; and providing an accumulator, the accumulator configured to be enabled or disabled based on the result of the adder; causing a comparison by said MAC unit of input values ​​within said window of input values.

13. The method of claim 12, wherein the MAC unit is configured to sequentially receive individual input values ​​within the window of input values ​​based on the MAC unit performing a max pooling operation.

14. The method of claim 13 , wherein the MAC unit is configured to identify a maximum value within the window of input values.

15. The method of claim 12, wherein the multiplexer is configured to select the result of the adder based on the MAC unit performing processing associated with a convolutional layer.

16. The method of claim 12, wherein a high bit of the result of the adder is used to enable or disable the accumulator.

17. 17. The method of claim 16, wherein the accumulator is configured to store the input value based on the input value being greater than a different value stored in the accumulator, and the accumulator is enabled based on a high bit associated with a result of the adder.

18. The method of claim 12 , wherein the accumulator is configured to be disabled so as not to store the input value based on the input value being less than a different value stored in the accumulator.

Images