Method of neural network processor performing multiplication operation and neural network processor

Abstract

A method of a neural network processor performing a multiplication operation and a neural network processor. In some embodiments, the method includes forming a first set of products and forming a second set of products. Forming the first set of products can include multiplying, in a first multiplier, a first activation value with a least significant subword and a most significant subword of a first weight to form a first partial product and a second partial product, and adding the first partial product and the second partial product. Forming the second set of products can include multiplying, in the first multiplier, a second activation value with a first subword and a second subword of a mantissa to form a third partial product and a fourth partial product, and adding the third partial product and the fourth partial product.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 112,271, filed November 11, 2020, entitled “SYSTEM AND METHOD FOR IMPROVING AREA AND POWER EFFICIENCY BY REDISTRIBUTING WEIGHT NIBBLES,” and U.S. Application No. 17 / 131,357, filed December 22, 2020, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] One or more aspects in accordance with embodiments of the present disclosure relate to processing circuits, and more specifically, to systems and methods for performing multiple sets of multiplications in a manner that accommodates outliers and is capable of performing both integer and floating point operations. BACKGROUND

[0003] Processors for neural networks can perform a large number of multiplication and addition operations, some of which can be inappropriately use processing resources because most of the numbers being processed can be relatively small and only a small fraction of outliers can be relatively large. Furthermore, some of the operations in such systems can be integer operations and some can be floating point operations, which can consume a large amount of chip area and power if performed on separate individual sets of dedicated hardware.

[0004] Accordingly, there is a need for systems and methods for performing multiple sets of multiplications in a manner that accommodates outliers and is capable of performing both integer and floating point operations. SUMMARY

[0005] According to embodiments of the application, there is provided a method comprising: forming a first set of products, each product in the first set of products being an integer product of a first activation value and a respective weight of a first plurality of weights; and forming a second set of products, each product in the second set of products being a floating point product of a second activation value and a respective weight of a second plurality of weights, the step of forming the first set of products comprising: multiplying, in a first multiplier, the first activation value and a least significant subword of a first weight of the first plurality of weights to form a first partial product, multiplying, in a second multiplier, the first activation value and a most significant subword of the first weight to form a second partial product, and adding the first partial product and the second partial product, the step of forming the second set of products comprising: multiplying, in a first multiplier, the second activation value and a first subword of a mantissa of a first weight of the second plurality of weights to form a third partial product, multiplying, in a second multiplier, the second activation value and a second subword of the mantissa to form a fourth partial product, and adding the third partial product and the fourth partial product.

[0006] In some embodiments, the second activation value is a half byte of a mantissa of the floating point activation value.

[0007] In some embodiments, a difference between a degree of effectiveness of the first subword of the mantissa and a degree of effectiveness of the second subword of the mantissa is equal to a width of the first subword of the mantissa.

[0008] In some embodiments, the step of adding the third partial product and the fourth partial product comprises performing offset addition in a first offset adder.

[0009] In some embodiments, an offset of the offset adder is equal to the width of the first subword of the mantissa.

[0010] In some embodiments, the step of adding the first partial product and the second partial product comprises performing offset addition in a first offset adder.

[0011] In some embodiments, the step of forming the first set of products further comprises: multiplying, in a third multiplier, the first activation value and a least significant subword of a second weight of the first plurality of weights to form a first partial product; multiplying, in the third multiplier, the first activation value and a most significant subword of the second weight to form a second partial product; and adding the first partial product and the second partial product.

[0012] In some embodiments, the step of forming the first set of products further comprises: multiplying, in a fourth multiplier, the first activation value and a least significant subword of a third weight of the first plurality of weights to form a first partial product, the third weight having a most significant half byte equal to zero; and adding the first partial product and zero.

[0013] In some embodiments, the first activation value is a most significant subword of an integer activation value.

[0014] In some embodiments, the method further comprises left-shifting the sum of the first partial product and the second partial product by a number of bits equal to a size of the first activation value.

[0015] According to embodiments of the invention, there is provided a system comprising: processing circuitry comprising a first multiplier, a second multiplier and a third multiplier, the processing circuitry being configured to: form a first set of products, each product in the first set of products being an integer product of a first activation value and a respective weight of a first plurality of weights, and form a second set of products, each product in the second set of products being a floating-point product of a second activation value and a respective weight of a second plurality of weights, the processing of forming the first set of products comprising: multiplying, in the first multiplier, the first activation value and a least significant subword of a first weight of the first plurality of weights to form a first partial product, multiplying, in the second multiplier, the first activation value and a most significant subword of the first weight to form a second partial product, and adding the first partial product and the second partial product, the processing of forming the second set of products comprising: multiplying, in the first multiplier, the second activation value and a first subword of a mantissa of a first weight of the second plurality of weights to form a third partial product, multiplying, in the second multiplier, the second activation value and a second subword of the mantissa to form a fourth partial product, and adding the third partial product and the fourth partial product.

[0016] In some embodiments, the second activation value is a half byte of a mantissa of a floating-point activation value.

[0017] In some embodiments, a difference between a significance of the first subword of the mantissa and a significance of the second subword of the mantissa is equal to a width of the first subword of the mantissa.

[0018] In some embodiments, the processing of adding the third partial product and the fourth partial product comprises performing offset addition in a first offset adder.

[0019] In some embodiments, an offset of the offset adder is equal to the width of the first subword of the mantissa.

[0020] In some embodiments, the processing of adding the first partial product and the second partial product comprises performing offset addition in a first offset adder.

[0021] In some embodiments, the process of forming the first set of products further comprises: multiplying, in a third multiplier, the first activation value with a least significant subword of a second weight of the first plurality of weights to form a first partial product; multiplying, in the third multiplier, the first activation value with a most significant subword of the second weight to form a second partial product; and adding the first partial product and the second partial product.

[0022] In some embodiments, the process of forming the first set of products further comprises: multiplying, in a fourth multiplier, the first activation value with a least significant subword of a third weight of the first plurality of weights to form a first partial product, the third weight having a most significant nibble equal to zero; and adding the first partial product and zero.

[0023] In some embodiments, the first activation value is a most significant subword of an integer activation value.

[0024] According to embodiments of the present invention, there is provided a system comprising: means for processing, the means for processing comprising a first multiplier, a second multiplier, and a third multiplier, the means for processing configured to: form a first set of products, each product in the first set of products being an integer product of a first activation value and a respective weight of a first plurality of weights, and form a second set of products, each product in the second set of products being a floating point product of a second activation value and a respective weight of a second plurality of weights, the process of forming the first set of products comprising: multiplying, in the first multiplier, the first activation value with a least significant subword of a first weight of the first plurality of weights to form a first partial product, multiplying, in the second multiplier, the first activation value with a most significant subword of the first weight to form a second partial product, and adding the first partial product and the second partial product, the process of forming the second set of products comprising: multiplying, in the first multiplier, the second activation value with a first subword of a mantissa of a first weight of the second plurality of weights to form a third partial product, multiplying, in the second multiplier, the second activation value with a second subword of the mantissa to form a fourth partial product, and adding the third partial product and the fourth partial product. BRIEF DESCRIPTION OF DRAWINGS

[0025] These and other features and advantages of the present disclosure will be appreciated and understood by reference to the following specification, claims, and drawings.

[0026] Figure 1 is a block diagram of a portion of a neural network processor according to embodiments of the present disclosure;

[0027] Figure 2A is a block diagram of a portion of a hybrid processing circuit according to embodiments of the present disclosure;

[0028] Figure 2Bis a data map according to embodiments of the present disclosure;

[0029] Figure 2C is a block diagram of a portion of a hybrid processing circuit according to embodiments of the present disclosure; and

[0030] Figure 3 is a data map according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0031] The detailed description set forth below, in connection with the appended drawings, is intended as a description of exemplary embodiments of a processor for fine-grain sparse integer and floating point operations provided in accordance with the present disclosure and is not intended to represent the only forms in which the present disclosure can be constructed or utilized. The description sets forth the features of the present disclosure in connection with the illustrated embodiments. It is to be understood, however, that the same or equivalent functions and structures can be accomplished by different embodiments that are also intended to be encompassed within the scope of the disclosure. As shown elsewhere herein, like element numbers are intended to indicate like or similar elements or features.

[0032] Neural networks (e.g., when performing inference) can perform a large number of computations in which activations (or “activation values”) (elements of an input feature map (IFM)) are multiplied by weights. The products of the activations and weights can form a multi-dimensional array, which can be summed along one or more axes to form an array or “tensor” (which can be referred to as an output feature map (OFM)). Referring to Figure 1 Specialized hardware can be employed to perform such computations. Activations can be stored in static random access memory (SRAM) 105 and fed into a multiplier accumulator (MAC) array, which can include (i) a plurality of blocks (which can be referred to as “bricks” 110), each of which can include a plurality of multipliers for multiplying an activation by a weight, (ii) one or more adder trees for adding together the products generated by the bricks, and (iii) one or more accumulators for accumulating the sums generated by the adder trees. Each activation value can be broadcast to a plurality of multipliers arranged conceptually in a row in Figure 1 The plurality of multipliers in the representation of FIG. 1 are conceptually arranged in a row. A plurality of adder trees 115 can be employed to form sums.

[0033] In operations, weights can fall in a range of values, and the distribution of values of weights is such that relatively small weights are significantly more common than relatively large weights. For example, if each weight is represented as an 8-bit number (e.g., an 8-bit integer, INT8, or an 8-bit floating point number), then many weights (e.g., a majority of the weights or more than 3 / 4 of the weights) can have a value less than 16 (i.e., the most significant nibble is zero); then, weights with a non-zero most significant nibble can be referred to as “outliers.” In some embodiments, properly constructed hardware can achieve improved speed and power efficiency by exploiting these characteristics of weights. However, the inventive concept is not limited as such, for example, each weight can also be represented as a 16-bit number (e.g., a 16-bit integer or a 16-bit floating point number, FP16).

[0034] Figure 2A A portion of the hybrid processing circuit (referred to as “hybrid” because it applies to both integer and floating point operations) is shown. Refer to Figure 2A In some embodiments, multiple multipliers 205 are used to multiply weights with activations, e.g., one nibble at a time. Each multiplier can be a 4x4 (i.e., 4-bit by 4-bit) multiplier, the first input 210 is configured to receive a respective weight nibble, and the second input 215 is configured to receive an activation nibble (which can be broadcast to all multipliers). An embodiment with nine multipliers is shown, in some embodiments, there are more multipliers (resulting in a more powerful but more expensive circuit), and in some embodiments, there are fewer multipliers (resulting in a less powerful but less costly circuit). A weight buffer 220 (only its output rows are shown) can include a respective register for each multiplier 205. The outputs of the multipliers 205 can be fed to multiple combination circuits 225, each combination circuit 225 can include one or more multiplexers 230, adders, and invert-and-shift circuits 235. The system can include the same number of combination circuits 225 as multipliers 205, or the system can contain fewer combination circuits 225 than multipliers 205 (as shown), or the system can contain more combination circuits 225 than multipliers 205.

[0035] In operations, each multiplier can produce one partial product during each clock cycle. These partial products can be added together to form integer products in integer operations (or partial products in floating point operations), each integer product can be processed by an invert-and-shift circuit 235, and the results (e.g., Unit0, Unitl, …, Unit7) can be sent to an adder tree (e.g., adder tree 0, adder tree 1, …, adder tree 7) to be added with other integer products. For example, in one example, as Figure 2AAs shown in the example, the first two values in the output row of the weight buffer 220 can be the least significant nibble L0 and the most significant nibble M0 of the first weight, respectively, and the activation value nibble being broadcast can be the least significant nibble of the first (8-bit) activation value. However, the inventive concepts are not limited thereto, e.g., in another example, the first two values in the output row of the weight buffer 220 can be the most significant nibble M0 and the least significant nibble L0 of the first weight, respectively. The activation value nibble can be multiplied with the least significant nibble L0 of the first weight to form the first partial product P0, and the activation value nibble can be multiplied with the most significant nibble M0 of the first weight to form the second partial product P1. Although Figure 2A The example shows that the first partial product P0 and the second partial product P1 are calculated using two multipliers (e.g., the first multiplier and the second multiplier) during one clock cycle, respectively, but the inventive concepts are not limited thereto, e.g., in another example, the first partial product P0 and the second partial product P1 can be calculated using the same multiplier (e.g., the third multiplier) during two clock cycles, respectively. In this example, the first partial product P0 can be calculated using the third multiplier during the first clock cycle, and the second partial product P1 can be calculated using the third multiplier during the second clock cycle. Figure 2A In the example, the output row of the weight buffer 220 can include the nibbles L2, L3, M3, M4, L7, M7, and 0, and the partial products P2, P3, …, P n-1 may be formed in a similar manner. These partial products can be routed to the first combination circuit 225 (the leftmost one among the combination circuits 225) through a connection fabric (or connection configuration) 240. Figure 2A The connection fabric 240 can include multiplexers (e.g., UL0, UM0, UL1, UM1, …, UL7, and UM7) 230 that perform data routing in the first combination circuit 225. Figure 2A The multiplexers 230 are drawn as separate elements in the example in order to facilitate the illustration of the data routing performed by the multiplexers 230 (using arrows). In the first combination circuit 225, the product of (i) the (two nibbles of the) weight and (ii) the activation value nibble can be calculated as the sum of the first partial product and the second partial product (by the corresponding offset adder 245).

[0036] As used herein, an "offset sum" of two values is the result of "offset addition," which is the sum of (i) a first of the two values and (ii) a second of the two values that is shifted left by a number of bits (e.g., four bits), and an "offset adder" is an adder that performs addition of two numbers with an offset between the positions of the significant bits of the two numbers, referred to as the "offset" of the offset adder. As used herein, the "significance" of a nibble (or, more generally, a sub-word (discussed in further detail below)) is the position in a word that the nibble occupies (e.g., whether the nibble is the most significant nibble or the least significant nibble of an 8-bit word). As such, the most significant nibble of an 8-bit word has a significance that is four bits greater than the least significant nibble. For example, for integer operations, the difference between the significance of a first sub-word of a weight and the significance of a second sub-word of the weight is equal to the width of the first sub-word of the weight, and the offset of the offset adder is equal to the width of the first sub-word of the weight. For example, for floating point operations, the difference between the significance of a first sub-word of a mantissa and the significance of a second sub-word of the mantissa is equal to the width of the first sub-word of the mantissa, and the offset of the offset adder is equal to the width of the first sub-word of the mantissa. The difference between the significance of a first sub-word and the significance of a second sub-word corresponds to an offset between the position in a word that the first sub-word occupies and the position in the word that the second sub-word occupies. A word (e.g., N bits, N being an integer greater than 1) can be divided into a plurality of sub-words. The sub-word of the plurality of sub-words that occupies the earliest position in the word is referred to as the most significant sub-word, and the sub-word of the plurality of sub-words that occupies the latest position in the word is referred to as the least significant sub-word.

[0037] Each invert-shift circuit 235 can convert between (i) a sign and magnitude representation and (ii) a two's complement representation, and each invert-shift circuit 235 can shift the result as needed for proper addition to occur in the adder tree. For example, if the active value nibble is the most significant nibble, the output of the offset adder 245 can be shifted (e.g., left shifted by 4 bits) so that the bits of the output will be properly aligned with the bits of other products (e.g., the product of the weight and the least significant nibble of the active value) in the adder tree. For example, if the multiplier 205 is an unsigned integer multiplier and the adder tree is a two's complement adder tree, the conversion between the sign and magnitude representation and the two's complement representation can be performed.

[0038] As shown in FIG. 2, the multiplier 205 can include a weight buffer 220, an activation buffer 230, and an adder tree 240. The weight buffer 220 can store a plurality of weight nibbles, and the activation buffer 230 can store a plurality of activation nibbles. The adder tree 240 can include a plurality of adders (e.g., offset adders) that add the plurality of weight nibbles and the plurality of activation nibbles to produce a plurality of products. The adder tree 240 can include a plurality of invert-shift circuits 235 that invert and shift the plurality of products to produce a plurality of output values. The output values can be provided to a summing circuit 250, which can sum the output values to produce a result. Figure 2B As shown in FIG. 2, the multiplier 205 can include a weight buffer 220, an activation buffer 230, and an adder tree 240. The weight buffer 220 can store a plurality of weight nibbles, and the activation buffer 230 can store a plurality of activation nibbles. The adder tree 240 can include a plurality of adders (e.g., offset adders) that add the plurality of weight nibbles and the plurality of activation nibbles to produce a plurality of products. The adder tree 240 can include a plurality of invert-shift circuits 235 that invert and shift the plurality of products to produce a plurality of output values. The output values can be provided to a summing circuit 250, which can sum the output values to produce a result.Figure 2B The blank cells in the diagram are shown. Preprocessing can rearrange these nibbles when filling the weight buffer (e.g., as shown by...). Figure 2B (As indicated by the arrow in the image), causing the weight buffer to contain a smaller proportion of zero-value nibbles than the original weight array 250. Figure 2B In the example, the eight weights (each consisting of the least significant nibble and the most significant nibble) are rearranged such that zero-value nibbles are discarded and non-zero nibbles are placed in eight positions of a row of the weight buffer (the ninth position contains zero), so that this row of the weight buffer consists of nine multipliers 205 ( Figure 2A When processing an array of multipliers, eight of the multipliers are used, and only one (the ninth) is not used. Therefore, as Figure 2B As shown, the most significant nibble of weight 2 in the original weight array 250 is 0, so the weight buffer only includes the least significant nibble L2 of weight 2 and does not include the most significant nibble of weight 2. In this case, the activation value nibble and the least significant nibble L2 of weight 2 can be multiplied in a multiplier 205 to form a partial product P2, and the partial product P2 and 0 can be fed into the combinational circuit 225 for addition. Furthermore, in some cases, the sparsity of the original weight array 250 may not be sufficient to allow all the most significant nibbles and their corresponding least significant nibbles to be in the same row of the weight buffer, and some or all products may be formed in two clock cycles, with the activation value remaining the same for both cycles. Preprocessing can also generate a control signal array, which can be used to control the connection structure 240 (e.g., multiplexer 230) such that each partial product is sent to the appropriate input of the offset adder 245 according to the validity of the factor that forms it.

[0039] like Figure 2C As shown, the hybrid processing circuit may further include multiple variable shift units (or "shift units", "variable shift circuits") 260, which enable the hybrid processing circuit to perform floating-point operations on floating-point activation and floating-point weights in floating-point mode. Each such floating-point number may be an FP16 floating-point number (using, for example, a format according to the IEEE 754-2008 standard) having a sign bit, an 11-bit mantissa (or "significant digit") (represented by a 10-bit and an implicit lead bit or "hidden bit"), and a five-bit exponent. The 11-bit mantissa may be padded with a zero and divided into three nibbles: a "high" (most significant) nibble, a "low" (least significant) nibble, and a (medium significant) "middle" nibble (so that concatenating the high, middle, and low nibbles in sequence produces a 12-bit (padded) mantissa).

[0040] The floating point multiplication can then be performed by the hybrid processing circuit of Figure 2C of the partial products of the "high, middle, and low half- bytes of the mantissa of each weight" and "the high, middle, and low half- bytes of the mantissa of the activation" by multiplying one half- byte of the weight with one half- byte of the activation at a time in each of the multipliers 205. The (12 bit (b) wide) output of each invert- shift circuit 235 can be fed to a corresponding variable shift unit 260, which in floating point mode can right shift its received data between 0 and N bits (where N can be 8, or a larger number, or a smaller number, depending in part on the size of the mantissa used in the adder tree (or the number of bits corresponding to that size), which can be selected based on the precision to be implemented), and which in integer mode can left shift its received data by 0 or M bits (where M can be the size of the most significant subword of the activation value used in the adder tree (or the number of bits corresponding to that size), e.g., when the activation value is the most significant subword of an integer activation value, the variable shift unit 260 in integer mode can left shift its received data by M bits; when the activation value is the least significant subword of an integer activation value, the variable shift unit 260 in integer mode can left shift its received data by 0 bits). Additional optional shifts can be obtained by selecting one or the other input of the offset adder 245 and by selecting the amount of shift applied in the invert- shift circuit 235. Thus, Figure 2C The hybrid processing circuit of

[0041] Figure 3 An example of pre-processing of an array of floating point weights is shown. In floating point representation, half- byte sparsity (which can be relatively common for integer weights, e.g., most of the weights have a zero value in the most significant half- byte) can be relatively rare, but most of the weights can be equal to zero, as shown for the original weight array 305, in which all three half- bytes (low (L), middle (M), and high (H)) of each of the three weights are zero. Figure 3The three nibbles showing the mantissa of each non-zero weight (weights 0, 2, 4, 5, and 6) can be rearranged as follows, first forming a first intermediate matrix 310, then forming a second intermediate matrix 315, and then forming a final matrix 320, which can fit in the weight buffer. In the final matrix, all non-zero elements are in the first two rows, and all products can be formed in two operations (e.g., in two clock cycles), whereas the original weight array 305 is loaded into the weight buffer, three operations would be used.

[0042] Although some examples are shown herein for embodiments with 8-bit weights, 8-bit activation values, a weight buffer that is four weights wide, and weights and activations that can be processed one nibble at a time, it will be understood that these parameters and other similar parameters in the present disclosure are used merely as specific, concrete examples for ease of explanation, and any of these parameters can be changed. As such, for example, the size of a weight can be a "word" and the size of a portion of a weight can be a "subword," in which Figure 2A In the embodiment of FIG. 3, the size of a word is one byte and the size of a subword is one nibble. In other embodiments, for example, a word can be 12 bits, a subword can be 6 bits, or a word can be 16 bits, a subword can be one byte. Furthermore, the methods of performing integer and floating point operations described in this application can be methods of a neural network processor to perform multiplication operations, and the components described in this application (e.g., SRAM, MAC, multiplier, combinatorial circuit, adder tree, multiplexer, adder, invert-shift circuit, variable shift unit, weight buffer, etc.) can be included in a neural network processor.

[0043] As used herein, a "portion" of something means "at least some" of the thing, which can mean less than all of the thing or can mean all of the thing. As a special case, a "portion" of a thing includes the entire thing (i.e., the entire thing is an example of a portion of the thing). As used herein, the term "or" is to be interpreted as "and / or", such that, for example, "A or B" means "A" or "B" or "A and B".

[0044] Each of the terms "processing circuitry" and "means for processing" as used herein represents any combination of hardware, firmware, and software for processing data or digital signals. The processing circuitry hardware can include, for example, an application specific integrated circuit (ASIC), a general purpose or special purpose central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and programmable logic devices such as a field programmable gate array (FPGA). In the processing circuitry, each function is performed by hardware configured (i.e., hardwired) to perform the function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. The processing circuitry can be fabricated on a single printed circuit board (PCB) or distributed among several interconnected PCBs. The processing circuitry can contain other processing circuitry, e.g., the processing circuitry can include two processing circuitries (FPGA and CPU) interconnected on a PCB.

[0045] As used herein, when a method (e.g., adjusting) or a first quantity (e.g., a first variable) is referred to as being "based on" a second quantity (e.g., a second variable), this means that the second quantity is an input to the method or affects the first quantity (e.g., the second quantity can be an input (e.g., the only input or one of several inputs) to a function that computes the first quantity, or the first quantity can equal the second quantity, or the first quantity can be the same as the second quantity (e.g., stored in the same location or locations in a memory)).

[0046] It will be understood that, although the terms "first", "second", "third", etc. can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, elements, components, regions, layers or sections discussed herein as first elements, first components, first regions, first layers or first sections can be called second elements, second components, second regions, second layers or second sections without departing from the spirit and scope of the inventive concept.

[0047] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concept. As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation and not as terms of degree, unless otherwise stated herein or in the claims, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of," when preceding a list of two or more items, cover all cases of that list, including individual items, combinations, and permutations, of the items in the list. In addition, use of the "of" in the articles "a" and "an" preceding code numbers in the description herein is by way of example and is not intended in a limiting sense. Furthermore, the use of the term "about" in the description herein is by way of example and is not intended in a limiting sense. As used herein, the term "use" and variations thereof can be considered synonymous with the term "utilize" and variations thereof.

[0048] It will be understood that when an element or layer is referred to as being "on" another element or layer, "connected to" another element or layer, "coupled to" another element or layer, or "adjacent to" another element or layer it can be directly on, directly connected to, directly coupled to, or directly adjacent to the other element or layer or one or more intervening elements or layers can be present. In contrast, when an element or layer is referred to as being "directly on," "directly connected to," "directly coupled to," or "immediately adjacent to" another element or layer, there are no intervening elements or layers present.

[0049] Any numerical range recited herein is intended to include all sub-ranges of the same numerical precision subsumed within the recited range. For example, a range of "1.0 to 10.0" or "1.0 to 10.0" is intended to include all sub-ranges, e.g., 1.0 to 2.3, 2.4 to 3.1, etc., within the same numerical precision. Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed therein and any minimum numerical limitation recited herein is intended to include all higher numerical limitations subsumed therein.

[0050] While exemplary embodiments for a processor for fine-grained sparse integer and floating point operations have been described and shown in detail herein, many modifications and variations will be readily apparent to those skilled in the art. Accordingly, it is to be understood that a processor for fine-grained sparse integer and floating point operations constructed in accordance with the principles of the present disclosure can be implemented differently than as specifically described herein. The invention is also defined in the following claims, and their equivalents.

Claims

1. A method of a neural network processor performing a multiplication operation, comprising: forming a first set of products, each product in the first set of products being an integer product of a first activation value and a respective weight of a first plurality of weights; and / or forming a second set of products, each product in the second set of products being a floating point product of a second activation value and a respective weight of a second plurality of weights, the step of forming the first set of products comprising: multiplying, in a first multiplier, the first activation value and a least significant subword of a first weight of the first plurality of weights to form a first partial product, multiplying, in a second multiplier, the first activation value and a most significant subword of the first weight of the first plurality of weights to form a second partial product, and adding the first partial product and the second partial product to form a product in the first set of products, the step of forming the second set of products comprising: multiplying, in a first multiplier, the second activation value and a first subword of a mantissa of a first weight of the second plurality of weights to form a third partial product, multiplying, in a second multiplier, the second activation value and a second subword of the mantissa to form a fourth partial product, and adding the third partial product and the fourth partial product to form a product in the second set of products.

2. The method of claim 1, wherein, the second activation value is a half word of a mantissa of a floating point activation value.

3. The method of claim 1, wherein, a difference between a degree of significance of the first subword of the mantissa and a degree of significance of the second subword of the mantissa is equal to a width of the first subword of the mantissa.

4. The method of claim 1, wherein, the step of adding the third partial product and the fourth partial product comprises performing an offset addition in a first offset adder.

5. The method of claim 4, wherein, an offset of the first offset adder is equal to the width of the first subword of the mantissa.

6. The method of claim 4, wherein, the step of adding the first partial product and the second partial product comprises performing an offset addition in a first offset adder.

7. The method of any one of claims 1 to 6, wherein, the step of forming the first set of products further comprises: multiplying, in a third multiplier, the first activation value and a least significant subword of a second weight of the first plurality of weights to form a fifth partial product; multiplying, in a third multiplier, the first activation value and a most significant subword of the second weight of the first plurality of weights to form a sixth partial product; and adding the fifth partial product and the sixth partial product to form a product in the first set of products.

8. The method of any one of claims 1 to 6, wherein, the step of forming the first set of products further comprises: multiplying, in a fourth multiplier, the first activation value and a least significant subword of a third weight of the first plurality of weights to form a seventh partial product, the third weight having a most significant half word equal to zero; and adding the seventh partial product and zero to form a product in the first set of products.

9. The method of any one of claims 1 to 6, wherein, the first activation value is a most significant subword of an integer activation value.

10. The method of claim 9, further comprising: the sum of the first partial product and the second partial product is shifted left by a number of bits equal to a size of the first activation value.

11. A neural network processor comprising: processing circuitry, the processing circuitry comprising a first multiplier and a second multiplier, the processing circuitry is configured to: form a first set of products, each product in the first set of products being an integer product of a first activation value and a respective weight of a first plurality of weights, and / or form a second set of products, each product in the second set of products being a floating point product of a second activation value and a respective weight of a second plurality of weights, The processing to form the first set of products includes: multiplying, in the first multiplier, the first activation value by a least significant subword of a first weight of the first plurality of weights to form a first partial product, multiplying, in the second multiplier, the first activation value by a most significant subword of the first weight of the first plurality of weights to form a second partial product, and adding the first partial product and the second partial product to form a product in the first set of products, The processing to form the second set of products includes: multiplying, in the first multiplier, the second activation value by a first subword of a mantissa of a first weight of the second plurality of weights to form a third partial product, multiplying, in the second multiplier, the second activation value by a second subword of the mantissa to form a fourth partial product, and adding the third partial product and the fourth partial product to form a product in the second set of products.

12. The neural network processor of claim 11, wherein, The second activation value is a half byte of a mantissa of a floating-point activation value.

13. The neural network processor of claim 11, wherein, A difference between a degree of significance of the first subword of the mantissa and a degree of significance of the second subword of the mantissa is equal to a width of the first subword of the mantissa.

14. The neural network processor of claim 11, wherein, The processing to add the third partial product and the fourth partial product includes performing an offset addition in a first offset adder of the processing circuitry.

15. The neural network processor of claim 14, wherein, An offset of the first offset adder is equal to the width of the first subword of the mantissa.

16. The neural network processor of claim 14, wherein, The processing to add the first partial product and the second partial product includes performing an offset addition in the first offset adder.

17. The neural network processor of any one of claims 11-16, wherein, The processing circuitry further comprises a third multiplier, and the processing to form the first set of products further includes: multiplying, in the third multiplier, the first activation value by a least significant subword of a second weight of the first plurality of weights to form a fifth partial product; multiplying, in the third multiplier, the first activation value by a most significant subword of the second weight of the first plurality of weights to form a sixth partial product; and adding the fifth partial product and the sixth partial product to form a product in the first set of products.

18. The neural network processor of any one of claims 11-16, wherein, The processing circuitry further comprises a fourth multiplier, and the processing to form the first set of products further includes: multiplying, in the fourth multiplier, the first activation value by a least significant subword of a third weight of the first plurality of weights to form a seventh partial product, the third weight having a most significant half byte equal to zero; and adding the seventh partial product and zero to form a product in the first set of products.

19. The neural network processor of any one of claims 11-16, wherein, The first activation value is a most significant subword of an integer activation value, The processing circuitry is further configured to left shift the sum of the first partial product and the second partial product by a number of bits equal to a size of the first activation value.

20. A neural network processor, comprising: An apparatus for processing, the apparatus for processing comprising a first multiplier, a second multiplier, and a third multiplier, The apparatus for processing is configured to: form a first set of products, each product in the first set of products being an integer product of a first activation value and a respective weight of a first plurality of weights, and / or form a second set of products, each product in the second set of products being a floating-point product of a second activation value and a respective weight of a second plurality of weights, The processing to form the first set of products includes: multiplying, in the first multiplier, the first activation value by a least significant subword of a first weight of the first plurality of weights to form a first partial product, multiplying, in a second multiplier, the first activation value with a most significant subword of a first weight of the first plurality of weights to form a second partial product, and adding the first partial product and the second partial product to form a product in a first product set, the process of forming a second product set includes: multiplying, in a first multiplier, the second activation value with a first subword of a mantissa of a first weight of the second plurality of weights to form a third partial product, multiplying, in a second multiplier, the second activation value with a second subword of the mantissa to form a fourth partial product, and adding the third partial product and the fourth partial product to form a product in a second product set.

Images