High-precision anchor implicit processing
HPAI numbers address the inefficiencies of floating point systems by embedding anchor values for dynamic scaling, enhancing precision and accuracy in data processing, particularly in neural networks.
Patent Information
- Application Number
- JP2023524302
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-10-27
- Filing Date
- 2021-10-21
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2041-10-21
AI Technical Summary
Existing data processing systems using floating point numbers, such as neural networks, face inefficiencies due to complex hardware and non-associative summation issues, particularly with formats like bfloat16, leading to varying results when numbers are added in different orders.
The implementation of High-Precision Anchored-Implicit (HPAI) numbers, which embed anchor values within operands to allow dynamic scaling and simplify accumulation processes, enabling faster and more accurate calculations by storing metadata and data in a single operand register.
HPAI numbers provide higher precision and accuracy in calculations, reducing hardware complexity and latency, and enabling efficient multi-lane accumulation without carry propagation, suitable for machine learning applications.
Smart Images

Figure 0007591140000001 
Figure 0007591140000002 
Figure 0007591140000003
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to data processing, and more particularly to data processing using High-Precision Anchored-Implicit (HPAI) numbers with embedded anchor values.
[0002] The use of floating-point representation is common in data processing systems such as neural networks and machine learning. A floating-point number includes a mantissa and an exponent that indicates the significance of the bits in the mantissa. Floating-point formats allow a wide range of numerical values to be represented using a finite number of bits. However, dedicated hardware that processes floating-point numbers is often complex and generally slower than equivalent hardware that processes integer-based numbers. To reduce hardware issues, the "Brain floating-point format with 16 bits" (bfloat16) format was created, which uses fewer bits than the single-precision IEEE-754 standard floating-point format. However, summation of floating-point numbers is non-associative, which means that different results are produced when floating-point numbers are added in different orders. For example, calculations involving bfloat16 values yield a 2 4 +2 20 -2 20 =(2 20 )-2 20 = 0. In contrast, 2 4 +(2 20 -2 20 )=2 4 +(0)=2 4 =16. Summary of the Invention
[0003] An apparatus is provided herein. The apparatus includes a processing circuit and a storage device. The processing circuit is configured to perform one or more processing operations in response to one or more instructions to generate an anchor data element. The storage device is configured to store the anchor data element. The format of the anchor data element includes an identification item, a duplicate item, and a data item. The data item is configured to hold a data value of the anchor data element. The identification item indicates an anchor value or one or more special values for the data value.
[0004] A method for controlling data processing is provided herein. The method includes controlling a processing circuit to perform one or more processing operations in response to one or more instructions to generate an anchor data element, and storing the anchor data element. The format of the anchor data element includes an identification item, a duplicate item, and a data item. The data item is configured to hold a data value of the anchor data element. The identification item indicates an anchor value or one or more special values for the data value.
[0005] A non-transitory computer-readable storage medium is provided herein that stores instructions for controlling data processing. The instructions, when executed by a processor, cause the processor to perform a plurality of operations. The operations include controlling a processing circuit to perform one or more processing operations in response to the instructions to generate an anchor data element and storing the anchor data element. The format of the anchor data element includes an identification item, a duplicate item, and a data item. The data item is configured to hold a data value of the anchor data element. The identification item indicates an anchor value or one or more special values for the data value.
[0006] The above and other features and advantages of the present disclosure will become readily apparent from the following detailed description of the best mode for carrying out the disclosure when considered in conjunction with the accompanying drawings. [Brief explanation of the drawings]
[0007] [Figure 1] 1 is a schematic diagram illustrating a data processing device for performing data processing operations under control of program instructions, in accordance with one or more exemplary embodiments. [Figure 2] 1 is a schematic diagram illustrating various representations of numerical values, according to one or more exemplary embodiments; [Figure 3] 1 is a schematic diagram of an anchor data element in accordance with one or more exemplary embodiments. [Figure 4] FIG. 10 is a schematic diagram illustrating the relationship between the range of values representable by single-precision floating-point values and the significance range of anchor data elements, in accordance with one or more exemplary embodiments. [Figure 5] FIG. 1 is a schematic diagram of an accumulator in accordance with one or more exemplary embodiments. [Figure 6] FIG. 10 is a schematic diagram of a first conversion of a bfloat16 number to an HPAI accumulator in accordance with one or more exemplary embodiments. [Figure 7] FIG. 10 is a schematic diagram of a second conversion of a bfloat16 number to an HPAI accumulator in accordance with one or more exemplary embodiments. [Figure 8] FIG. 10 is a schematic diagram of a conversion data path from a bfloat16 value to an anchor data element in accordance with one or more exemplary embodiments. [Figure 9] FIG. 1 is a schematic diagram of a single-lane HPAI accumulation data path in accordance with one or more exemplary embodiments. [Figure 10] FIG. 10 is a schematic diagram of a multi-lane HPAI accumulation sequence, according to one or more exemplary embodiments. [Figure 11] FIG. 10 is a schematic diagram of another accumulator in accordance with one or more exemplary embodiments. [Figure 12] FIG. 1 is a schematic diagram of a two-lane dot-four product circuit according to one or more exemplary embodiments. [Figure 13] FIG. 10 is a schematic diagram of a first alternative anchor data element, according to one or more exemplary embodiments. [Figure 14] FIG. 10 is a schematic diagram of a second alternative anchor data element, according to one or more exemplary embodiments. [Figure 15]1A-1C are schematic diagrams of circuit layout synthesis results for a floating-point dot-product circuit and an HPAI dot-product circuit, in accordance with one or more exemplary embodiments. DETAILED DESCRIPTION OF THE INVENTION
[0008] Embodiments of the present disclosure generally provide an improved representation for High-Precision Anchored (HPA) numbers that allows metadata to be stored in the same operand register as the data. Storing the metadata and data in a single operand register enables HPA-type multiply-accumulate instructions. The improvement modifies each operand to include an indication of the anchor value for the corresponding data. The improved representation may be referred to as a High-Precision Anchored Implicit (HPAI) number and / or anchor data element.
[0009] The identifier portion of the operand functions as a "lane identifier" that enables several features for the accumulator circuit that improve processing. The identifier portion allows the data in the operand to be dynamically scaled as additional input values are accumulated. The dynamic scaling feature allows the accumulation to continue without external intervention, eliminating the programmer's task of specifying the anchor value in advance and the requirement to adjust the anchor value as the accumulation requires. The operand has an implicit anchor value that is adjusted as input values are accumulated.
[0010] The improved format retains a redundant bit that allows multi-lane accumulation without carry propagation. The redundant bit increases accumulation throughput when there is no change to the lane identification and when there is a change to the lane identification. The accumulation hardware is simplified by embedding an indication of the anchor value for the data item in the operand. The simplified hardware can perform the accumulation and modify the lane identification value when the accumulation is updated to a higher numeric range.
[0011] Various embodiments of the data items enable fast and simple hardware for calculating dot products, which are often used in machine learning applications. The internal calculation logic is independent of physical size and is not limited by the result data size. This provides internal calculations with higher precision than the final result, increasing the accuracy of the result to meet the precision requirements of the application, and further allows the calculation result to be of any size, including 16 bits, 32 bits, 64 bits, or more.
[0012] Referring to Figure 1, a schematic diagram illustrating a data processing device 90 for performing data processing operations under control of program instructions is shown, in accordance with one or more exemplary embodiments. Data processing device 90 includes a storage element 92 and a processor core (or apparatus) 100. Storage element 92 is configured to store data 94 and program instructions 96. Processor core 100 includes a storage device 102, processing circuitry 104, an instruction fetch unit 106, an instruction pipeline 108, and a decoder 110. It will be understood that in practice data processing device 90 may include many additional elements, and that the representation in Figure 1 is simplified to aid understanding.
[0013] Data signals (e.g., D) are exchanged between storage element 92 and processor core 100. Data signals D carry data 94 to be processed by processor core 100 per program instruction 96 and result values produced by the processing. Instruction signals (e.g., I) are transferred from storage element 92 to processor core 100. Instruction signals I carry program instructions 96 to be executed by processor core 100.
[0014] The storage element 92 is implemented as one or more mass storage devices. The storage element 92 is coupled to the processor core 100. The storage element 92 is operable to store data 94 and program instructions 96. In various embodiments, the storage element 92 includes one or more memory cards, one or more solid-state drives, and / or one or more hard disk drives configured to communicate with the processor core 100 via data signals D and instruction signals I. Other forms of storage may be implemented within the storage element 92 to meet the design criteria of a particular application.
[0015] The processor core 100 is operable to execute program instructions 96 to manipulate the data 94. The processing circuitry 104 is configured to perform one or more processing operations in response to one or more of the program instructions 96 to generate and store anchor data elements. The format of the anchor data element includes an identification item, a duplicate item, and a data item. The data item is configured to hold the data value of the anchor data element. The identification item indicates an anchor value or one or more special values for the data value.
[0016] In operation, program instructions 96 are fetched from storage elements 92 by instruction fetch unit 106 and provided to instruction pipeline 108. When program instructions 96 reach the appropriate stage in instruction pipeline 108, they are decoded by decoder 110, which generates control signals that function to control the operation of storage device 102 and processing circuitry 104 to perform the processing operation(s) specified by the decoded program instruction. Multiple input operands are read from storage device 102 and provided to processing circuitry 104, which operates on the input operands and may write one or more subsequent result values back into storage device 102.
[0017] The storage device 102 may be implemented in a variety of different forms. In various embodiments, the storage device 102 is implemented as a register bank. The operands may include, for example, floating-point operands, fixed-point operands, integer operands, HPA operands, and / or HPAI operands. The storage device 102 may function to store a mixture of these types of operands, depending on the configuration of the storage device 102. The operands may have different levels of precision, as may be predefined by their format or programmably specified using metadata associated with the registers. Further details of high precision anchor implicit operands may be found in U.S. Pat. No. 10,579,338, which is incorporated herein by reference in its entirety.
[0018] Referring to FIG. 2, a schematic diagram illustrating various exemplary representations of numeric values is shown, in accordance with one or more exemplary embodiments. A single-precision floating-point operand 120 is formed by a sign, an exponent, and a mantissa. The single-precision floating-point operand 120 has a predetermined size (e.g., [31:0]) and can represent values having a wide variety of magnitudes, as indicated by their exponent values. The precision with which a number can be represented is limited by the size of the mantissa. Floating-point operations are generally more complex and slower than integer operations.
[0019] Also shown is a 32-bit integer operand 122. The integer operand 122 is a predefined size (e.g., [31:0]) with a range of 0 to 2 for unsigned integers. 32 Represents numbers in the range -1 to -2 for signed integers 31 ~2 31 Integer arithmetic can represent numbers in the range -1. Integer arithmetic is generally faster (compared to floating-point operations) and consumes less energy to perform, but suffers from the disadvantage of a more restricted range of numbers that can be specified compared to the range of numbers that can be represented by floating-point values.
[0020] Additionally, a high precision anchor implicit (HPAI) operand is shown. The HPAI operand in this example includes a vector of multiple (three in this example) anchor data elements 124a-124c, each containing a 32-bit integer (e.g., [31:0]) in this example. The HPAI operand has associated metadata. The metadata includes anchor values embedded within one or more of the anchor data elements 124a-124c that indicate bit significance of components forming part of the HPAI number. The anchor value(s) directly or indirectly specify a lower bit significance bound and an upper bit significance bound. The term metadata, as used herein, may be considered to correspond to a data value(s) and may be used to specify bit significance of the HPAI number. Different components may be used together to specify bit values that contiguously span a range of bit significance. Depending on the locations of the lower bit significance bound and the upper bit significance bound, the range of bit significance may include the location of the binary point. It is also possible that the binary point location is outside the range of bit significance specified for a particular precision anchor implied value.
[0021] The anchor value(s) may be provided to be able to represent a range of bit significances extending from the minimum significance that can be represented by a floating-point value (e.g., a single-precision FP value) to the maximum bit significance that can be represented by that floating-point value.
[0022] The number of components that form the high precision anchor implicit number may vary between different implementations. The size of the components is fixed in some embodiments, but may vary in other embodiments. In some embodiments, the overall width of the bit significance range may be constrained to vary in units of a fixed component size (e.g., for 16-bit components, the bit significance range may have widths of, for example, 16, 32, 64, 128, 192, 256, etc.). It is also possible for the width of the bit significance range to vary continuously in 1-bit-wide steps.
[0023] By embedding the anchor value(s) within the anchor data element, generally, the programmer is spared the manual task of setting and updating the significance (e.g., the anchor value) of the corresponding HPAI value. The anchor value can specify bit significance in a variety of different ways. One example is specifying the bit significance of the lower bound of each vector component. Thus, each component of a vector may include an integer value representing the portion of the valid bits of the value within the overall range of bit significance, and metadata (anchoring) representing the significance of the least significant bit within that component. Another option is that the anchor value(s) specify the lower bound of the bit significance of the entire high-precision anchor implicit number, along with the full width of the range of bit significance. A further option is that the anchor value(s) include data specifying the lower and upper bounds of the range representing bit significance. Additionally, variations are possible such as the anchor value(s) including the lower bound of the range of bit significance together with the number of components known to be of a fixed-width component.
[0024] Referring to FIG. 3, a schematic diagram of an exemplary implementation of an anchor data element 124 according to one or more exemplary embodiments is shown. The anchor data element 124 includes a data item 130, a duplicate item 132, and an identification item 134. The identification item 134 includes a lane item 136 and a special item 138. In various embodiments, the anchor data element 124 can be represented as an N-bit (e.g., 32-bit) two-valued string.
[0025] The data item 130 is configured to store the data value of the anchor data element 124. The data item 130 is a D-bit (e.g., 16-bit) field within the anchor data element 124, where D < N. In various embodiments, the data item 130 occupies the lower D bits of the anchor data element 124.
[0026] The duplicate item 132 is configured to provide extra bits within the anchor data element 124 to store data with higher precision than can be held by the data item 130 alone. In some embodiments, the duplicate item 132 is an O-bit (e.g., 11-bit) field within the anchor data element 124, where O < N. The duplicate item 132 is used during addition / accumulation operations such that the data within the data item 130 would otherwise overflow the boundaries of the data item 130.
[0027] The lane item 136 is configured to store a lane identification value. In various embodiments, the lane item 136 is an L-bit (e.g., 4-bit) field having an identification item 134, where L < N. The lane identification value within the lane item 136 is an anchor value for the data values held within the data item 130 and the duplicate item 132. The lane identification value provides metadata that establishes the bit-significance window of the data values.
[0028] The special item 138 is configured to store a bit (or flag) that determines when the remaining bits within the identification item 134 are either an identification value or one of several possible special values. The special item 138 is an S-bit (e.g., 1-bit) field having an identification item 134, where S < N. In various embodiments, while the special item 138 holds a normal value (e.g., a logical 0 value), the remaining bits within the identification item 134 are interpreted as a lane identification value. While the special item 138 holds a predetermined value (e.g., a logical 1 value), the remaining bits within the identification item 134 specify that the anchor data element 124 is one of positive infinity, negative infinity, not-a-number, and an overflow indication. Other combinations of the L bits within the identification item 134 can be used to represent other special values that meet the design criteria of a particular application.
[0029] By combining items 130-134 into a single N-bit element, the N-bit element may be loaded and stored in a single N-bit storage element (or register). Thus, processing circuitry 104 of Figure 1 is configured to access identification item 134, duplicate item 132, and data item 130, which are together in a single addressable location.
[0030] Referring to Figure 4, a schematic diagram illustrating the relationship between the range of values representable by a single-precision floating-point value and an exemplary significance range of the anchor data element 124 of Figure 3 is shown, in accordance with one or more exemplary embodiments. For single-precision floating-point numbers, the range 140 of bit values that may be specified is generally approximately 2 +127 ~about 2 -149 The bit significance window 142 of the anchor data element 124 may be anywhere within the range 140, but may cover only a portion of the range 140. The bit significance window 142 may be programmable by a programmer to establish an initial range. The bit significance window 142 may also be automatically adjusted by the hardware operating the anchor data element 124 when the data value held therein exceeds the upper limit of the window 142.
[0031] Referring to FIG. 5, a schematic diagram of an exemplary 288-bit HPAI accumulator 150 is shown, in accordance with one or more exemplary embodiments. HPAI accumulator 150 can be viewed as a 17-lane accumulator (e.g., 15, 14, ..., 0, -1) spanning the range [+128:-142]. Each lane covers 16 bits of accumulation. Lane 15 generally covers accumulation [128:113], lane 14 covers accumulation [112:97], and so on. Lane 1 is optionally used for the least significant bits of products with very small exponents (e.g., ≦-113). An implicit 11-bit overlap of adjacent lanes (not shown) exists when the overlap item 132 of one lane extends into an adjacent lane.
[0032] Mapping a bfloat16 number into HPAI accumulator 150 involves determining a lane identification value for the corresponding anchor data element 124. For 16 lanes, the lane identification value may be obtained as the four most significant bits of the bfloat exponent (e.g., exponent[7:4]). Thus, the bfloat16 product may be shifted right by ~exp[3:0] (e.g., ~exp[3:0] == 15 (decimal) ~exp[3:0]) to be placed in the correct lane of HPAI accumulator 150. In the illustrated example, the lane identification value is 0 (decimal).
[0033] Referring to Figure 6, a schematic diagram of a first exemplary conversion 160 of a bfloat16 number to an HPAI accumulator is shown in accordance with one or more exemplary embodiments. -4 An example value of ×1.e01e results in an 8-bit bfloat16 exponent of 0x7B (hex). The four most significant bits of the exponent (e.g., 0x7) determine the lane identification value. A mantissa of 0xf00f (hex) is combined with the exponent to return an HPAI number corresponding to a 32-bit value of 0x38000f00 (hex). Data item 130 of anchor data element 124 (HPAI) can hold a 16-bit fraction (e.g., 0x0f00). Thus, 12 bits (e.g., 0xf00) from the original mantissa 0xf00f are left padded to 16 bits (e.g., 0x0f00) and assigned to lane 7, and the last 4 bits of the original mantissa (e.g., 0xf) are assigned to lane 6. For a one-lane conversion, the value of lane 6 is discarded (e.g., truncated). A negative product will result in the destination overlap region being set to all ones (not shown).
[0034] 7, a schematic diagram of a second exemplary conversion 170 of a bfloat16 number to an HPAI accumulator is shown, in accordance with one or more exemplary embodiments. +4Assume an exemplary value of x1.e01e, an 8-bit bfloat16 exponent of 0x83 (hex). The four most significant bits of the exponent (e.g., 0x8) determine the lane identification value. A mantissa of 0xf00f (hex) is combined with the exponent to return an HPAI number corresponding to a 32-bit value of 0x4000000f (hex). Data item 130 of anchor data element 124 (HPAI) can hold a 16-bit fraction (e.g., 0x000f). Thus, 4 bits (e.g., 0xf) from the original mantissa 0xf00f are left padded to 16 bits (e.g., 0x000f) and assigned to lane 8, and the last 12 bits of the original mantissa (e.g., 0x00f) are assigned to lane 7. For a one-lane conversion, the value in lane 7 is discarded (e.g., truncated).
[0035] 8, a schematic diagram of an exemplary conversion data path 180 from a bfloat16 value to an anchor data element 124 is shown, in accordance with one or more exemplary embodiments. Data path 180 may be implemented within processor core 100. An input bfloat16 (BF) product 182 may be received by data path 180. Data path 180 includes blocks 184, 186, 188, 190, 192, and 194.
[0036] The bfloat16 product is parsed into an exponent, a mantissa, and a "SIZD" field in block 184. The SIZD field indicates one or more of a sign, infinity, zero, and not-a-number indicator. The exponent field is parsed into four most significant bits (e.g., [7:4]) and four least significant bits (e.g., [3:0]) in block 186.
[0037] In block 188, the SIZD field and the four most significant exponent bits are used to determine both the special value stored in the special field 138 and the lane identification value stored in the lane field 136 of the anchor data element 124. In block 190, the mantissa of the 16-bit bfloat16 product and the +sign bit of the SIZD field are converted to a 17-bit two's complement number. Incrementing the two's complement number can be deferred until a later addition / accumulation. Bit
[16] of the HPAI number is sign-extended to the entire overlap area in data item 130 before accumulating. Bit
[16] of the converted product specifies whether to increment or not. In block 192, the four least significant bits of the bfloat16 exponent field are used to shift the two's complement number to the 16-bit size of data item 130. In various embodiments, the order of the two's complement block 190 and the shifter block 192 may be reversed.
[0038] In block 194, identification item 134 and data item 130 are populated with the values determined by blocks 188, 190, and 192. Duplicate item 132 may be populated with all 0 bits or all 1 bits, depending on the sign of the converted bfloat16 product. At the end of data path 180, bfloat16 product 182 is converted into anchor data element 124 suitable for use by processing circuitry 104 and / or elsewhere in processor core 100.
[0039] 9, a schematic diagram of an exemplary single-lane HPAI accumulation data path 200 is shown in accordance with one or more exemplary embodiments. Data path 200 may be implemented within processor core 100. Data path 200 includes blocks 202, 204, 206, 208, 210, 212, and 214. The illustrated example data path 200 generally adds 5+17-bit HPAI bfloat products to 5+27-bit HPAI accumulator products.
[0040] In block 202, the bfloat16 product recently converted to anchor data element 124 is parsed into 17 bits of new data (e.g., HPAI bfloat) and a 5-bit identification value (e.g., ID1). In block 204, the accumulated data currently in the accumulator may be parsed into 27 bits of accumulated data (e.g., HPAI Acc) and a 5-bit identification value (e.g., ID0). A single 5-bit comparison is performed in block 206 where the identification value ID0 is compared to the identification value ID1. The result of the comparison, taking into account special values, is the identification value (ID result) = max(ID1, ID0). The ID result may be presented to blocks 208, 210, and 214.
[0041] In block 208, a selection (e.g., 2 x 27-bit AND gates) is performed to determine whether the new data has a non-zero value or a zero value. The selection is based on the ID result and the new data. To account for the total overlap region, the selection logic of block 208 sign-extends the new data from 17 bits to 27 bits. In block 210, a selection (e.g., 2 x 27-bit AND gates) is performed to determine whether the accumulate data has a non-zero value or a zero value. The selection is based on the ID result and the accumulate data. In block 212, the results of the two selection blocks 208 and 210 are added. The adder can be a 27-bit integer adder. The adder may include carry-in capability for completion of HPAI bfloat two's complement values.
[0042] In response to detecting an overflow of the 27-bit accumulation data, an overflow signal (e.g., ovf) may be asserted by block 212 to block 214. Block 214 responds to the overflow signal assertion by forcing the ID result to a special overflow value.
[0043] 10, a schematic diagram of an exemplary multi-lane HPAI accumulation sequence 220 is shown, in accordance with one or more exemplary embodiments. Sequence 220 may be implemented within processor core 100. Sequence 220 includes multiple lanes 222a-222g that accumulate new data in a destination HPAI element 224. Sequence 220 includes steps 226, 228, 230, 232, 234, 236, 238, and 240.
[0044] During the accumulation addition, if ID(op0)==ID(op1) (e.g., block 206 in FIG. 9), the HPAI elements are added; otherwise, the HPAI element with the larger ID value is used. The lane ID (anchor value) of the resulting value may be updated in the destination HPAI element 224 during the accumulation. For reproducible summation, the lane ID is limited to increase during the accumulation. The rules for handling special values in the new bfloat16 data may follow the IEEE standard.
[0045] The first new data (e.g., BF#1) may be received by the accumulator in step 226. BF#1 resides in lanes 222d and 222e (e.g., lanes 4 and 5). Therefore, the accumulator adds BF#1 to the destination HPAI element 224 in step 228, along with an identification value indicating the highest lane number (e.g., 5). The value of BF#1 may be stored in data item 130 of the destination HPAI element 224.
[0046] Subsequent new data (e.g., BF#2) may be received by the accumulator in step 230. BF#2 resides in lanes 222c and 222d (e.g., lanes 3 and 4). During the comparison of the discrimination values, the accumulator determines that BF#1 is significantly greater than BF#2, so in step 232, the accumulator leaves the resulting value in the destination HPAI element 224 unchanged.
[0047] Another new data (e.g., BF#3) is received by the accumulator in step 234. BF#3 resides in lanes 222d and 222e (e.g., lanes 4 and 5). During the comparison of the discrimination values, the accumulator determines that BF#1 and BF#3 have the same magnitude. Therefore, the accumulator adds BF#3 to BF#1 already stored in the result value in step 236. In this example, adding BF#3 to BF#1 does not result in an overflow, so the discrimination value in the destination HPAI element 224 remains in lane 5.
[0048] Another new data (e.g., BF#4) is received at the accumulator in step 238. BF#4 resides in lane 222f (e.g., lane 6). During a comparison of the lane identification values, the accumulator determines that BF#4 has a higher lane identification value (e.g., 6) than the lane identification value of the accumulated data (e.g., 5). Because BF#4 is significantly greater than BF#1+BF#3, the accumulator replaces the result value in the destination HPAI element 224 with BF#4 in step 240 and automatically updates the identification value to 6. The sequence 220 can continue adding more new data until all of the data is summed or an overflow occurs. If an overflow occurs, the accumulation operation can continue, but the result is retained as a special value representing "overflow."
[0049] 11 , a schematic diagram of an exemplary implementation of accumulator 260 is shown in accordance with one or more exemplary embodiments. Accumulator 260 may be implemented within processor core 100. Accumulator 260 includes a first vector storage element 262, a second vector storage element 264, a product value 266, multiple floating-point multipliers 268, multiple bfloat16-to-HPAI conversion units 270, multiple HPAI adders 272, and a result vector storage element 274. The initial values being accumulated may be multiple floating-point values (e.g., multiple bfloat16 values).
[0050] A plurality of (e.g., eight) initial values are stored in a first vector storage element 262, and a plurality of (e.g., eight) additional initial values are stored in a second vector storage element 264. Pairs of initial values from the first vector storage element 262 and the second vector storage element 264 are arranged as a product value 266.
[0051] The floating-point multiplier 268 generates a plurality of first intermediate values by multiplying the first intermediate values by the initial product value 266. The bfloat16-to-HPAI conversion unit 270 converts the first intermediate values to a plurality of second intermediate values. Each of the second intermediate values has the format of the anchor data element 124 (e.g., FIG. 3). The HPAI adder 272 generates a corresponding result value by 3:1 accumulation of two second intermediate values and one locally accumulated value. Each result value has the format of the anchor data element 124. Once the accumulation is complete, the result value may be loaded into the result vector storage element 274.
[0052] 12, a schematic diagram of an exemplary two-lane dot-four product circuit 280 is shown, in accordance with one or more exemplary embodiments. The dot-four product circuit 280 may be implemented within processor core 100. The dot-four product circuit 280 includes a first vector storage element 282, a second vector storage element 284, a third vector storage element 286, multiple pairs of floating-point multipliers 288a-288b, multiple pairs of bfloat16-to-HPAI converters 289a-289b, multiple pairs of HPAI adders 290a-290b, multiple pairs of HPAI accumulators 292a-292b, and a result vector storage element 294. The initial values held in the first vector storage element 282 and the second vector storage element 284 may be multiple floating-point values (e.g., multiple bfloat16 values).
[0053] The dot-four product circuit 280 generally calculates the dot-four product as follows:
[0054] D[0]+=(Zn[0]*Zm[0])+(Zn[1]*Zm[1])+(Zn[2]*Zm[2])+(Zn[3]*Zm[3])
[0055] D[1]+=(Zn[4]*Zm[4])+(Zn[5]*Zm[5])+(Zn[6]*Zm[6])+(Zn[7]*Zm[7])
[0056] The pairs of initial values held in the first vector storage element 282 and the second vector storage element 284 are parsed into a third vector storage element 286. Floating-point multipliers 288a-288b generate a plurality of first intermediate values by multiplying a first set of initial values and generate a plurality of second intermediate values by multiplying a second set of initial values. Subsequently, HPAI conversion circuits 289a-289b convert the first intermediate values into a plurality of third intermediate values and convert the second intermediate values into a plurality of fourth intermediate values. The third intermediate values and the fourth intermediate values have the format of anchor data element 124.
[0057] Using data in the anchor data element format, HPAI adder 290a generates a plurality of fifth intermediate values by accumulating the upper portions of the third and fourth intermediate values. HPAI adder 290b generates a plurality of sixth intermediate values by accumulating the lower portions of the third and fourth intermediate values. The fifth and sixth intermediate values have the format of anchor data element 124.
[0058] HPAI accumulator 292a generates a plurality of first result values by accumulating the fifth intermediate value. HPAI accumulator 292b generates a plurality of second result values by accumulating the sixth intermediate value. The first result values and the second result values have the format of anchor data element 124. The first result value is loaded into the high portion of result vector storage element 294. The second result value is loaded into the low portion of result vector storage element 294.
[0059] The dot-four product circuit 280 generally solves the problem of dealing with 32-bit dynamically scaled HPAI result values that contain only less significant bits. The two portions of the 64-bit result value contain two 32-bit HPAI lanes. Furthermore, the result value can use a shared common HPAI lane identification value. The dot-four product circuit 280 performs approximately eight multiplications per 128 bits per cycle, thus having similar performance to an equivalent bfloat16 dot-four product circuit, but with higher precision and better accuracy in neural network applications.
[0060] Referring to FIG. 13, a schematic diagram of an exemplary first alternative anchor data element 124d is shown, in accordance with one or more exemplary embodiments. The first alternative anchor data element 124d includes a single identification item 134 and multiple (e.g., two shown) sets 300 and 302 of duplicate / data items. Each set 300 and 302 includes one of the duplicate items and one of the data items. Each set 300 and 302 provides 39 bits for data (16 bits) and corresponding duplicates (13 bits). The first alternative anchor data element 124d provides a 64-bit, two-lane HPAI element. As the identification value increments, the lower-order duplicate / data bits may be replaced by the higher-order duplicate / data bits, and the original lower-order duplicate / data bits are dropped.
[0061] Referring to FIG. 14, a schematic diagram of an exemplary second alternate anchor data element 124e is shown, in accordance with one or more exemplary embodiments. The second alternate anchor data element 124e includes a single identification item 134, a single duplicate item 132, and multiple (e.g., three as shown) data items 130a-130c. Each data item 130a-130c provides 16 bits of data. The second alternate anchor data element 124e generally provides a 64-bit, three-lane HPAI element. In response to an increment of the identification value, the duplicate and data bits may be shifted right by 16 bits, with the least significant 16 bits being dropped. This scheme eliminates the repeatability property of the accumulation in exchange for retaining a larger number of result bits.
[0062] 15, a schematic diagram of an exemplary circuit layout synthesis result for floating-point dot-product circuit 320 and HPAI dot-product circuit 340 is shown, in accordance with one or more exemplary embodiments. The inputs of both circuits 320 and 340 are designed to receive bfloat16 values. Floating-point dot-product circuit 320 includes a set of 16-bit floating-point multipliers 322, a 25-bit floating-point remainder rounding and zero-clearing adder 324, a 32-bit floating-point remainder rounding and zero-clearing adder 326, and an additional flop 328. HPAI dot-product circuit 340 includes a set of 16-bit floating-point multipliers 342 and a floating-point to HPAI conversion, adder, and accumulate circuit 344.
[0063] Synthesis results show that HPAI dot-product circuit 340 is approximately 2.3 times smaller than floating-point dot-product circuit 320. HPAI dot-product circuit 340 implements single-cycle execution of bfloat16 to HPAI conversion, product sum, and accumulation. bfloat16 multipliers 322 and 342 are essentially the same in both circuits 320 and 340. HPAI dot-product circuit 340 eliminates the product-exponent comparator.
[0064] HPAI accumulate circuit 344 is approximately three times smaller than the 25-bit floating-point / 32-bit floating-point adder pair 324 and 326. In particular, HPAI accumulate circuit 344 occupies approximately 54% of the size of a single fully IEEE-compliant 32-bit floating-point adder (e.g., 808 micrometers square with 228 flops). HPAI convert, adder, and accumulate circuit 344 provides single-cycle execution of convert, sum of products, and accumulate. Accumulate circuit 344 has two bfloat16 product carry-save adders, and the adders lack pipeline flops.
[0065] bfloat operations support the use of "implicit" metadata within anchor data elements. For example, a 25-bit floating-point product (e.g., 1-bit sign, 8-bit exponent, 16-bit fraction) is 2 -126 ~2 +127 Therefore, bfloat operations can accommodate all possible products in a full-width "Kulisch" accumulator (acc[127:-141] = 269 bits). The four most significant bits of the bfloat exponent may be used as a lane identification value (or anchor value) embedded in anchor data element 124. The embedded lane identification (anchor) value eliminates the manual task for the programmer to specify metadata and enables three-register multiply (floating-point)-accumulate (HPAI) instructions.
[0066] The anchor data element format provides an HPAI adder that is significantly smaller and has lower latency (e.g., fewer pipeline flops) than a 32-bit floating-point adder. The anchor data element representation can support dot-product instructions (multiply-accumulate). Furthermore, HPAI versions of the bfloat instructions have been successfully coded and integrated into TensorFlow, an open-source software library developed by the Google Brain team.
[0067] In one or more embodiments of the apparatus, the processing circuitry is further configured to generate a result value by accumulating the multiple data values in the multiple anchor data elements, and to automatically modify an anchor value associated with the result value during the accumulation.
[0068] In one or more embodiments of the apparatus, the processing circuitry is further configured to generate a plurality of first intermediate values by multiplying the plurality of initial values, convert the plurality of first intermediate values to a plurality of second intermediate values, each of the plurality of second intermediate values having a format of the anchor data element, and generate a result value by accumulating the plurality of second intermediate values, the result value having the format of the anchor data element.
[0069] In one or more embodiments of the apparatus, the initial values are floating point values.
[0070] In one or more embodiments of the apparatus, the processing circuit generates a plurality of first intermediate values by multiplying a first set of a plurality of initial values, generates a plurality of second intermediate values by multiplying a second set of a plurality of initial values, converts the plurality of first intermediate values to a plurality of third intermediate values, each of the plurality of third intermediate values having a format of an anchor data element, converts the plurality of second intermediate values to a plurality of fourth intermediate values, each of the plurality of fourth intermediate values having a format of an anchor data element, and generates a plurality of fifth intermediate values by accumulating a high order portion of the plurality of third intermediate values and the plurality of fourth intermediate values. the plurality of sixth intermediate values by accumulating lower portions of the plurality of third intermediate values and the plurality of fourth intermediate values, each of the plurality of sixth intermediate values having the format of the anchor data element; generating a first result value by accumulating the plurality of fifth intermediate values, the first result value being in the format of the anchor data element; and generating a second result value by accumulating the plurality of sixth intermediate values, the second result value being in the format of the anchor data element.
[0071] In one or more embodiments of the apparatus, the first set of initial values and the second set of initial values are floating point values.
[0072] In one or more embodiments of the apparatus, the format of the anchor data element includes a single identifying item, a plurality of overlapping items, and a plurality of data items arranged into a plurality of sets, each of the plurality of sets including one of the plurality of overlapping items and one of the plurality of data items.
[0073] In one or more embodiments of the apparatus, the format of the anchor data element includes a single identifying item, a single overlapping item, and multiple data items concatenated together.
[0074] In one or more embodiments of the apparatus, the processing circuitry is further configured to store the identification items, the duplicate items, and the data items together in a single addressable location.
[0075] In one or more embodiments of the device, the one or more special values include positive infinity, negative infinity, not a number, or overflow.
[0076] In one or more embodiments, the method further includes generating a result value by accumulating a plurality of data values in a plurality of anchor data elements, and automatically modifying an anchor value associated with the result value during the accumulation.
[0077] In one or more embodiments, the method further includes generating a plurality of first intermediate values by multiplying the plurality of initial values; converting the plurality of first intermediate values to a plurality of second intermediate values, each of the plurality of second intermediate values having a format of the anchor data element; and generating a resultant value by accumulating the plurality of second intermediate values, the resultant value having a format of the anchor data element.
[0078] In one or more embodiments, the method includes generating a plurality of first intermediate values by multiplying a first set of a plurality of initial values; generating a plurality of second intermediate values by multiplying a second set of a plurality of initial values; converting the plurality of first intermediate values to a plurality of third intermediate values, each of the plurality of third intermediate values having a format of an anchor data element; converting the plurality of second intermediate values to a plurality of fourth intermediate values, each of the plurality of fourth intermediate values having a format of an anchor data element; and generating a plurality of fifth intermediate values by accumulating upper portions of the plurality of third intermediate values and the plurality of fourth intermediate values. generating a plurality of sixth intermediate values by accumulating lower portions of the plurality of third intermediate values and the plurality of fourth intermediate values, wherein each of the plurality of sixth intermediate values has the format of the anchor data element; generating a first resultant value by accumulating the plurality of fifth intermediate values, wherein the first resultant value is in the format of the anchor data element; and generating a second resultant value by accumulating the plurality of sixth intermediate values, wherein the second resultant value is in the format of the anchor data element.
[0079] In one or more embodiments of the method, the format of the anchor data element includes a single identifying item, a plurality of overlapping items, and a plurality of data items arranged into a plurality of sets, each of the plurality of sets including one of the plurality of overlapping items and one of the plurality of data items.
[0080] In one or more embodiments of the method, the format of the anchor data element includes a single identifying item, a single overlapping item, and multiple data items concatenated together.
[0081] In one or more embodiments, the method further includes storing the identifying item, the duplicate item, and the data item together in a single addressable location.
[0082] In one or more embodiments of the non-transitory computer-readable storage medium, the operations further include generating a result value by accumulating a plurality of data values in a plurality of anchor data elements, and automatically modifying an anchor value associated with the result value during the accumulation.
[0083] In one or more embodiments of the non-transitory computer-readable storage medium, the operations further include generating a plurality of first intermediate values by multiplying a plurality of initial values; converting the plurality of first intermediate values to a plurality of second intermediate values, each of the plurality of second intermediate values having a format of the anchor data element; and generating a resultant value by accumulating the plurality of second intermediate values, the resultant value having a format of the anchor data element.
[0084] While the best mode for carrying out the present disclosure has been described in detail, those skilled in the art to which the present disclosure pertains will recognize various alternative designs and embodiments for carrying out the present disclosure within the scope of the appended claims.
Claims
1. 1. An apparatus comprising: a processing circuit configured to perform one or more processing operations in response to one or more instructions to generate an anchor data element; a storage device configured to store the anchor data element, the format of the anchor data element includes an identification item, a duplication item, and a data item; the data item is configured to hold a data value of the anchor data element; a storage device, the identification item indicating an anchor value or one or more special values for the data value; An apparatus comprising:
2. The processing circuitry comprises: generating a result value by accumulating a plurality of said data values in a plurality of said anchor data elements; and automatically modifying the anchor value associated with the result value during the accumulation; The apparatus of claim 1 , further configured to:
3. The processing circuitry comprises: generating a plurality of first intermediate values by multiplying a plurality of initial values; converting the plurality of first intermediate values into a plurality of second intermediate values, each of the plurality of second intermediate values having the format of the anchor data element; and generating a result value by accumulating said plurality of said second intermediate values, said result value having said format of said anchor data element; The apparatus of claim 1 , further configured to:
4. The apparatus of claim 3 , wherein the initial values are floating point values.
5. The processing circuitry comprises: generating a plurality of first intermediate values by multiplying a first set of a plurality of initial values; generating a plurality of second intermediate values by multiplying the second set of the plurality of initial values; converting the plurality of first intermediate values into a plurality of third intermediate values, each of the plurality of third intermediate values having the format of the anchor data element; converting the plurality of second intermediate values into a plurality of fourth intermediate values, each of the plurality of fourth intermediate values having the format of the anchor data element; generating a plurality of fifth intermediate values by accumulating upper portions of the plurality of third intermediate values and the plurality of fourth intermediate values, each of the plurality of fifth intermediate values having the format of the anchor data element; generating a plurality of sixth intermediate values by accumulating lower portions of the plurality of third intermediate values and the plurality of fourth intermediate values, each of the plurality of sixth intermediate values having the format of the anchor data element; generating a first result value by accumulating the plurality of fifth intermediate values, the first result value being in the format of the anchor data element; and generating a second resultant value by accumulating the plurality of sixth intermediate values, the second resultant value being in the format of the anchor data element; The apparatus of claim 1 , further configured to:
6. 6. The apparatus of claim 5, wherein the first set of the plurality of initial values and the second set of the plurality of initial values are a plurality of floating point values.
7. 2. The apparatus of claim 1, wherein the format of the anchor data element includes a single identifying item, a plurality of overlapping items, and a plurality of data items arranged into a plurality of sets, each of the plurality of sets including one of the plurality of overlapping items and one of the plurality of data items.
8. The apparatus of claim 1 , wherein the format of the anchor data element includes a single identifying item, a single overlapping item, and a plurality of the data items concatenated together.
9. The apparatus of claim 1 , wherein the processing circuitry is further configured to store the identification items, the duplicate items, and the data items together in a single addressable location.
10. The apparatus of claim 1 , wherein the one or more special values include positive infinity, negative infinity, not-a-number, or overflow.
11. 1. A method for controlling data processing, comprising the steps of: controlling a processing circuit to perform one or more processing operations in response to the one or more instructions to generate an anchor data element; storing the anchor data element, the format of the anchor data element includes an identification item, a duplication item, and a data item; the data item is configured to hold a data value of the anchor data element; the identification item indicates an anchor value or one or more special values for the data value; and A method comprising:
12. generating a result value by accumulating a plurality of said data values in a plurality of said anchor data elements; automatically modifying the anchor value associated with the result value during the accumulation; The method of claim 11 further comprising:
13. generating a plurality of first intermediate values by multiplying a plurality of initial values; converting the plurality of first intermediate values into a plurality of second intermediate values, each of the plurality of second intermediate values having the format of the anchor data element; generating a result value by accumulating the plurality of the second intermediate values, the result value having the format of the anchor data element; The method of claim 11 further comprising:
14. generating a plurality of first intermediate values by multiplying a first set of a plurality of initial values; generating a plurality of second intermediate values by multiplying a second set of the plurality of initial values; converting the plurality of first intermediate values into a plurality of third intermediate values, each of the plurality of third intermediate values having the format of the anchor data element; converting the plurality of second intermediate values into a plurality of fourth intermediate values, each of the plurality of fourth intermediate values having the format of the anchor data element; generating a plurality of fifth intermediate values by accumulating upper portions of the plurality of third intermediate values and the plurality of fourth intermediate values, each of the plurality of fifth intermediate values having the format of the anchor data element; generating a plurality of sixth intermediate values by accumulating lower portions of the plurality of third intermediate values and the plurality of fourth intermediate values, each of the plurality of sixth intermediate values having the format of the anchor data element; generating a first result value by accumulating the plurality of fifth intermediate values, the first result value being in the format of the anchor data element; generating a second resultant value by accumulating the plurality of sixth intermediate values, the second resultant value being in the format of the anchor data element; The method of claim 11 further comprising:
15. 12. The method of claim 11, wherein the format of the anchor data element includes a single identifying item, a plurality of overlapping items, and a plurality of data items arranged into a plurality of sets, each of the plurality of sets including one of the plurality of overlapping items and one of the plurality of data items.
16. The method of claim 11 , wherein the format of the anchor data element includes a single said identifying item, a single said overlapping item, and a plurality of said data items concatenated together.
17. storing said identification item, said duplicate item, and said data item together in a single addressable location; The method of claim 11 further comprising:
18. 1. A non-transitory computer-readable storage medium storing instructions for controlling data processing, the instructions, when executed by a processor, causing the processor to: controlling a processing circuit to perform one or more processing operations in response to said instructions to generate an anchor data element; storing the anchor data element, the format of the anchor data element includes an identification item, a duplication item, and a data item; the data item is configured to hold a data value of the anchor data element; the identification item indicates an anchor value or one or more special values for the data value; and A non-transitory computer-readable storage medium that causes a plurality of operations to be performed, including:
19. The plurality of operations: generating a result value by accumulating a plurality of said data values in a plurality of said anchor data elements; automatically modifying the anchor value associated with the result value during the accumulation; 20. The non-transitory computer-readable storage medium of claim 18, further comprising:
20. The plurality of operations: generating a plurality of first intermediate values by multiplying a plurality of initial values; converting the plurality of first intermediate values into a plurality of second intermediate values, each of the plurality of second intermediate values having the format of the anchor data element; generating a result value by accumulating the plurality of the second intermediate values, the result value having the format of the anchor data element; 20. The non-transitory computer-readable storage medium of claim 18, further comprising:
Citation Information
Patent Citations
Lane position information for vector processing
JP2018533799A
Transforming Anchor Data Elements
JP2022519259A
Handling overflow or underflow of anchor data values
JP2022519848A
