Hierarchical shared exponential floating-point data type

Abstract

Embodiments of the present disclosure include systems and methods for providing a hierarchical shared-exponent floating-point data type. First and second shared exponent values ​​are determined based on the exponent values ​​of the plurality of floating-point values. A third shared exponent value is determined based on the first shared exponent value and the second shared exponent value. First and second difference values ​​are determined based on the first shared exponent value, the second shared exponent value and the third shared exponent value. A sign value and a mantissa value are determined for the plurality of floating-point values. The sign value and mantissa value, the third shared exponent value, the first difference value and the second difference value for each floating-point value of the plurality of floating-point values ​​are stored in a data structure for the shared-exponent floating-point data type.

Description

Technical Field

[0001] Background

[0001] This disclosure relates to computing hardware. More specifically, this disclosure relates to floating-point data types.

Background Art

[0002]

[0002] Neural networks are machine learning models used for a variety of different applications (e.g., image classification, computer vision, natural language processing, speech recognition, handwritten character recognition, etc.). Neural networks can be trained for a specific purpose by running a dataset through the entire neural network, comparing the results from the neural network to known results, and updating the network based on the difference.

[0003]

[0003] Efficient training of neural networks and inference with low-fidelity data types using neural networks may require developing a data type that maximizes the fidelity of each bit while minimizing the computing cost. This can be formulated as an optimization problem whose objective is to maximize the quantization signal-to-noise ratio (QSNR) metric while minimizing the area overhead of the hardware dot product unit.

Summary of the Invention

Means for Solving the Problems

[0004] Brief Description of the Drawings

[0004] Various embodiments of the present disclosure are shown by way of example and not limitation in the figures of the accompanying drawings.

Brief Description of the Drawings

[0005] [Figure 1]

[0005] A computing system according to some embodiments is shown. [Figure 2]

[0006] Examples of converting floating-point values ​​to hierarchical shared exponential floating-point data types are shown in several embodiments. [Figure 3]

[0007] The following are exemplary data structures for storing the hierarchical shared exponential floating-point data type shown in Figure 2, according to several embodiments. [Figure 4]

[0008] Here are some other examples of converting floating-point values ​​to hierarchical shared exponential floating-point data types, according to several embodiments. [Figure 5]

[0009] The following are exemplary data structures for storing the hierarchical shared exponential floating-point data type shown in Figure 4, according to several embodiments. [Figure 6]

[0010] Examples of multiplying hierarchical shared exponential floating-point data type variables are shown in several embodiments. [Figure 7]

[0011] The process for creating hierarchical shared exponential floating-point data type variables is shown using several embodiments. [Figure 8]

[0012] A simplified block diagram of an exemplary computer system in several embodiments is shown. [Figure 9]

[0013] Several embodiments of neural network processing systems are shown. [Modes for carrying out the invention]

[0006] Detailed explanation

[0014] The following description includes numerous examples and specific details for illustrative purposes to provide a more detailed understanding of the disclosure. Such examples and details should not be construed as unreasonably limiting the elements of the claims or the claimed subject matter as a whole. It will be apparent to those skilled in the art that, based on different claim languages, the claimed subject matter may include some or all of the features of these examples, either individually or in combination, and may further include variations and equivalents of the features and techniques described herein.

[0007]

[0015] Here, we describe techniques for providing hierarchical shared exponential floating-point data types. In some embodiments, a computing system can be configured to create and manage data for hierarchical shared exponential floating-point data types. For example, a computing system can create data for a hierarchical shared exponential floating-point data type from several numbers that have been converted as needed to floating-point numbers (e.g., half-precision floating-point numbers, single-precision floating-point numbers, double-precision floating-point numbers, etc.). To achieve this, the computing system first determines the exponential value of each floating-point number. Next, the computing system quantizes these exponential values ​​based on a hierarchy of shared exponential values. Then, the computing system quantizes the mantissa of each floating-point number based on the quantized exponential values. Finally, the computing system stores the hierarchy of shared exponential values ​​and the quantized mantissa in a data structure for the hierarchical shared exponential floating-point data type.

[0008]

[0016] In some embodiments, efficient training / inference of ultra-narrow bit widths (e.g., mantissas of 1-4 bits) may require developing data types that maximize the fidelity of each bit while minimizing computing costs. This can be formulated as an optimization problem, the objective of which is to maximize the quantized signal-to-noise ratio (QSNR) while minimizing the area overhead of hardware dot product units. As will be discussed, the hierarchical shared exponential floating-point data types described herein can push the Pareto frontier of the QSNR multiplicative cumulative (MAC) area further than other floating-point data types, thereby extracting greater efficiency. In this ultra-narrow bit width regime, such an increase in effective QSNR can directly lead to better achievable accuracy for both training neural networks and inference using those neural networks.

[0009]

[0017] In some embodiments, the data types described herein are shared exponent data types. That is, the exponents are shared among a small number of elements. The number of elements can be referred to as the tile size or block size. The use of a small tile size may not be the best choice from a hardware perspective because, as the number of mantissa bits is reduced, most of the cost of the integer multiplier is eliminated and the accumulator begins to occupy most of the total area of the multiply-accumulator (MAC) unit. Therefore, by adopting a relatively large tile size, the accumulator silicon cost can be amortized while avoiding excessive mapping of elements within the tile to zero due to the difference between their private and shared exponents.

[0010]

[0018] In some embodiments, the data types described herein are intended to provide a balance between the allowable tile size and the coding efficiency in an ultra-narrow bitwidth regime. The coding efficiency can be defined by the following equation.

[0011]

[0019]

Equation

[0012]

[0020] Where

Equation

[0013]

[0021] In some embodiments, a hierarchical shared exponent floating-point data type decomposes each tile into nested sub-tiles such that the shared exponent of each sub-tile can differ from the next level of sub-tiles by only a scale value. These scales can be implemented as right shifts of an adder tree that is unfolded in a dot product unit.

[0014]

[0022] In some embodiments, the data types described herein have several adjustable parameters. One parameter is the tile size, which is the number of elements sharing a global exponent. Another parameter is the number of mantissa bits. In some embodiments, the mantissa values are represented using two's complement, and in other embodiments, the mantissa values are represented using a sign / magnitude format. The number of mantissa bits can take on any value of one or more.

[0015]

[0023] Another parameter is the representation of the shared exponent. In some embodiments, the shared exponent can be represented using a single level. In the case of a single-level representation, one parameter is the subtile size, which is the number of elements that share a local scale to adjust the global exponent. Each tile can be divided into multiple subtiles of equal size m, each subtile sharing a common subscale. When m=1, a right shifter can be added to the leaves of the adder tree in a dot product unit. The larger the value of m, the closer the right shifter is to the root of the adder tree. For example, when m=16, the right shifter is added to the fourth level of the adder tree, Log2(l6)=. Another parameter of the single-level representation is the subtile scale, which specifies the amount of right shift allowed per element. The scale encodes the maximum allowed difference between the global shared exponent and the exponent selected within the subtile. For example, a scale of 1 in the leaves of the adder tree means that each input element can be shifted by 0 or 1, resulting in a 2-bit combination multiplexing. In some cases, the maximum index per tile can be used as the global shared index. Therefore, the scale is always a positive number greater than or equal to zero. In some embodiments, other index selection techniques can be used. In cases where the global shared index is not the maximum index of a tile, the selected scale can be a positive or negative value.

[0016]

[0024] In other embodiments, the shared exponent is represented hierarchically. Instead of using a large single-level shift to scale the shared exponent, the shift is distributed across multiple levels of the adder tree, and the scale per shift is instead constrained. One parameter of the hierarchical representation is the hierarchy level, which is the number of hierarchy levels of the adder tree used to adjust the shared exponent. Indexing starts from the leaves of the tree. For example, with a tile size of 32, there will be a maximum of 5 levels in the adder tree. 3 hierarchy levels implies the use of right shifts within the first 3 levels, starting from the leaves of the adder tree. Another parameter of the hierarchical representation is the subtile scale, which is the maximum scale per element allowed at each level. The scale encodes the difference with the shared exponent (not necessarily the global exponent) at the next level. The shared exponent may or may not be the maximum exponent. For example, if the maximum and minimum exponents of a tile are calculated, the difference between the maximum and minimum gives the maximum scale difference. If the hierarchy level is set to 5, the upper limit of this exponential difference (maximum-minimum) can be 5 (because there are 5 levels of subtiles), and the global exponent can be set to (max-min).clip(5) / 2. This divides the tile into subtiles hierarchically, and at each level, it is checked whether a scale of -1 / 0 / 1 must be set for each subtile based on the subtile's exponent.

[0017]

[0025] The accumulator bit width parameter specifies the data type used for accumulating at the end of the adder tree. Examples of such data types include half-precision floating-point data types, single-precision floating-point data types, double-precision floating-point data types, etc. In some embodiments, the default data type may be a single-precision floating-point value. The truncation bit width may be another parameter. This parameter specifies the amount of truncation after each integer addition, thereby effectively reducing area overhead. An example of an effective truncation scheme is to truncate the output of the integer multiplier at the leaves of the adder tree to an 8-bit mantissa, so that the bit width of the output of the integer adder at each level of the adder tree increases by 1. Such a technique can reduce area overhead while minimizing the impact on the achievable QSNR (i.e., precision), for example.

[0018]

[0026] Figure 1 shows a computing system 100 according to several embodiments. As shown, the computing system 100 includes a hierarchical shared exponential floating-point data type (HASEFPDT) manager 105, an exponent manager 110, a mantissa manager 115, a numeric data storage 120, and a floating-point data storage 125. The numeric data storage 120 stores numeric data that can be converted to hierarchical shared exponential floating-point data types. Examples of numeric data include floating-point numbers, integers, etc. The floating-point data storage 125 is configured to store hierarchical shared exponential floating-point data types. Examples of such data types are described in detail below. In addition, the floating-point data storage 125 can store definitions for hierarchical shared exponential floating-point data types.

[0019]

[0027] The HASEFPDT manager 105 is responsible for managing hierarchical shared exponential floating-point data types and the data stored according to such data types. For example, the HASEFPDT manager 105 can create data to be stored as a hierarchical shared exponential floating-point data type by retrieving the definition for the hierarchical shared exponential floating-point data type from the floating-point data storage 125. Next, the HASEFPDT manager 105 can retrieve numeric data from the numeric data storage 120 and convert the numeric data into floating-point numbers (e.g., half-precision floating-point numbers, single-precision floating-point numbers, double-precision floating-point numbers, etc.). The number of numeric data to be retrieved by the HASEFPDT manager 105 is determined based on the definition for the hierarchical shared exponential floating-point data type. The HASEFPDT manager 105 then sends a request to the exponential manager 110 to determine the converted numeric data, the definition for the hierarchical shared exponential floating-point data type, and the shared exponential value for the numeric data. When the HASEFPDT manager 105 receives a shared exponent value from the exponent manager 110, the HASEFPDT manager 105 sends a request to the mantissa manager 115 to determine the numeric data, the shared exponent value, the definition for the hierarchical shared exponent floating-point data type, and the mantissa for the numeric data. When the HASEFPDT manager 105 receives the mantissa from the mantissa manager 115, the HASEFPDT manager 105 stores the mantissa and the shared exponent value in a data structure configured to store data for the hierarchical shared exponent floating-point data type.

[0020]

[0028] The HASEFPDT manager 105 can also read values ​​from data of a hierarchical shared exponential floating-point data type. For example, the HASEFPDT manager 105 can receive requests to read a specific value from data stored according to a hierarchical shared exponential floating-point data type (e.g., a data structure for storing data according to a hierarchical shared exponential floating-point data type) (e.g., from a component running on computing system 100, from a component running on another computing system). In response, the HASEFPDT manager 105 retrieves a dummy value of the requested value from the data of the hierarchical shared exponential floating-point data type, determines an exponential value for the requested value based on the shared exponential value stored in the data of the hierarchical shared exponential floating-point data type, and generates the requested value based on the dummy value and the exponential value.

[0021]

[0029] The exponent manager 110 is configured to determine exponent values ​​for hierarchical shared exponential floating-point data types. For example, the exponent manager 110 may receive a request from the HASEFPDT manager 105 for numeric data, a definition for a hierarchical shared exponential floating-point data type, and a request to determine exponent values ​​for the numeric data. In response, the exponent manager 110 determines the hierarchy of exponent values ​​based on the numeric data and the definition for the hierarchical shared exponential floating-point data type. The exponent manager 110 then transmits the hierarchy of exponent values ​​to the HASEFPDT manager 105.

[0022]

[0030] The mantissa manager 115 handles the determination of mantissa values ​​for hierarchical shared exponential floating-point data types. For example, the mantissa manager 115 can receive requests from the HASEFPDT manager 105 for numeric data, a shared exponential value, a definition for a hierarchical shared exponential floating-point data type, and a mantissa value for the numeric data. Based on the shared exponential value and the definition for the hierarchical shared exponential floating-point data type, the mantissa manager 115 determines the mantissa value for the numeric data. The mantissa manager 115 then sends the mantissa value to the HASEFPDT manager 105.

[0023]

[0031] Figure 2 shows examples of converting floating-point values ​​to a hierarchical shared exponential floating-point data type in several embodiments. Specifically, Figure 2 shows Table 200, which contains various values ​​throughout the process of converting floating-point values ​​to a hierarchical shared exponential floating-point data type. This example is illustrated by referring to Figure 1. In this example, the four values ​​in row 205 are converted to a hierarchical shared exponential floating-point data type that stores shared exponential values ​​using a hierarchy of shared exponential values ​​having three levels. As shown, the four values ​​are presented as decimal values. The HASEFPDT manager 105 begins by retrieving the definition for the hierarchical shared exponential floating-point data type from the floating-point data storage 125. Next, the HASEFPDT manager 105 retrieves the values ​​for row 205 from the numeric data storage 120 and converts them to floating-point values ​​(e.g., half-precision floating-point values, single-precision floating-point values, double-precision floating-point values, etc.). Then, the HASEFPDT manager 105 sends a request to the exponent manager 110 to determine the floating-point values, the definitions for the hierarchical shared exponent floating-point data types, and the exponent values ​​for the floating-point values.

[0024]

[0032] Upon receiving definitions and requests for floating-point values, hierarchical shared exponential floating-point data types, the exponential manager 110 determines the exponential value for each floating-point value, as depicted in row 210 of Table 200. These exponential values ​​are considered Level 0 exponential values, as shown in row 215 in this example. The exponential manager 110 then determines the Level 1 shared exponential value for each pair of exponential values. In this example, the exponential manager 110 determines the Level 1 shared exponential value by determining the exponential value that has the highest value for each pair of exponential values ​​and using it as the Level 1 shared exponential value. As shown in row 220, the highest exponential value among the exponential values ​​for the first pair of values ​​is -3. The highest exponential value among the exponential values ​​for the second pair of values ​​is -2. The exponential manager 110 then determines the Level 1 scale value for each pair of Level 0 exponential values ​​and the corresponding Level 1 shared exponential value. Here, the exponent manager 110 determines the scale value for level 1 by determining the difference between the exponent value for level 0 and the corresponding shared exponent value for level 1. If the difference is greater than 1, the exponent manager 110 approximates the scale value for level 1 to 1.

[0025]

[0033] For the first and second pairs of Level 0 exponential values ​​and corresponding Level 1 shared exponential values, and for the second value, the exponential manager 110 determines that each of the Level 1 scale values ​​is 0 (i.e., the difference between -3 and -3 is 0). For the third pair of Level 0 exponential values ​​and corresponding Level 1 shared exponential values, the exponential manager 110 determines that the difference between the Level 0 exponential value and the Level 1 shared exponential value is 2. Since this is greater than 1, the exponential manager 110 approximates the Level 1 scale value to 1. For the fourth pair of Level 0 exponential values ​​and corresponding Level 1 shared exponential values, the exponential manager 110 determines that the Level 1 scale value is 0 (i.e., the difference between -2 and -2 is 0). The Level 1 scale values ​​are depicted in row 225 of Table 200.

[0026]

[0034] After determining the Level 1 shared exponent and scale values, the exponent manager 110 determines the Level 2 shared exponent. In this example, the exponent manager 110 determines the Level 2 shared exponent based on the Level 1 shared exponent. Specifically, the exponent manager 110 determines the Level 2 shared exponent by determining the exponent that has the highest value among the Level 1 shared exponents and using it as the Level 2 shared exponent. Here, the exponent manager 110 determines that the Level 2 shared exponent is -2, as shown in line 230.

[0027]

[0035] Next, the exponent manager 110 determines the Level 2 scale value for each pair of Level 1 shared exponent values ​​and corresponding Level 2 shared exponent values. In this example, the exponent manager 110 determines the Level 2 scale value by determining the difference between the Level 1 shared exponent value and the corresponding Level 2 shared exponent value. If the difference is greater than 1, the exponent manager 110 approximates the Level 2 scale value to 1. For the first pair of Level 1 shared exponent values ​​and corresponding Level 2 shared exponent values, the exponent manager 110 determines that the Level 2 scale value is 1 (i.e., the difference between -2 and -3 is 1). For the second pair of Level 1 shared exponent values ​​and corresponding Level 2 shared exponent values, the exponent manager 110 determines that the Level 2 scale value is 0 (i.e., the difference between -2 and -2 is 0). The Level 2 scale values ​​are shown in row 235. Row 240 depicts the total scale value for each floating-point value, which is the sum of the Level 1 scale value and the Level 2 scale value. Row 245 shows the total shared exponential scale value for each floating-point value, which is the exponential value determined for each floating-point value by subtracting the respective total scale value from the Level 2 shared exponential value. Row 250 depicts the difference between the determined exponential value and the original exponential value for each floating-point value. Row 255 shows the quantized magnitude of the mantissa for the floating-point value. In this example, a 2-bit mantissa is used, with the first bit used for the sign (not shown in Table 200) and the second bit used for the magnitude. Row 260 shows the quantized exponential value for each floating-point value, which is the same as the shared exponential scale value in row 245. Finally, line 265 shows the decimal representation of the quantized floating-point value after converting the original floating-point value to a hierarchical shared exponential floating-point data type.

[0028]

[0036] After the exponent manager 110 determines the Level 2 shared exponent, Level 2 scale value, and Level 1 scale value, it sends them to the HASEFPDT manager 105. The HASEFPDT manager 105 then sends a request to the mantissa manager 115 to determine the floating-point value in line 205, the Level 2 shared exponent, the Level 2 scale value, the Level 1 scale value, the definition for the hierarchical shared exponent floating-point data type, and the mantissa value for the floating-point value. Upon receiving the data and request, the mantissa manager 115 determines the mantissa value for each floating-point value. If the exponent difference (shown in line 250) is greater than 0, the mantissa manager 115 quantizes the magnitude of the mantissa for each floating-point value to 0. Otherwise, the mantissa manager 115 uses the available bits (one bit for sign and one bit for magnitude in this example) and their respective quantized exponent values ​​(shown in line 260) to best approximate the original floating-point value. Here, the mantissa manager 115 determines the quantized magnitude values ​​for each of the floating-point values ​​to be the values ​​depicted in line 255. The mantissa manager 115 then sends the determined mantissa values ​​to the HASEFPDT manager 105.

[0029]

[0037] When the HASEFPDT manager 105 receives a mantissa from the mantissa manager 115, the HASEFPDT manager 105 stores the mantissa, a level 2 shared exponent, a level 2 scale value, and a level 1 scale value in a data structure configured to store data for a hierarchical shared exponent floating-point data type. Figure 3 shows an exemplary data structure 300 for storing the hierarchical shared exponent floating-point data type shown in Figure 2, according to several embodiments. The data structure 300 can be generated based on a definition for the hierarchical shared exponent floating-point data type, where the definition for the data type specifies four 2-bit mantissa values ​​for four floating-point values, four 1-bit level 1 scale values, two 1-bit level 2 scale values, and an 8-bit level 2 shared exponent value. As shown, the data structure 300 includes four attributes: a mantissa attribute, a level 1 scale attribute, a level 2 scale attribute, and a shared exponent attribute. The mantissa attribute stores a 2-bit mantissa value determined for each floating-point value. The Level 1 scale attribute stores the 1-bit Level 1 scale value determined for each floating-point value. The Level 2 scale value attribute stores the 1-bit Level 2 scale value determined for each pair of floating-point values. The shared exponent attribute value stores the 8-bit Level 2 shared exponent value determined for the floating-point value. In this example, the Level 2 shared exponent value is represented using a bias of 127. Therefore, in this example, the value 125 (i.e., 127 + (-2)) is stored in the shared exponent attribute.

[0030]

[0038] Figure 4 shows another example of converting floating-point values ​​to a hierarchical shared exponential floating-point data type in several embodiments. Specifically, Figure 4 shows Table 400, which contains various values ​​throughout the process of converting floating-point values ​​to different hierarchical shared exponential floating-point data types. This example is also illustrated by referring to Figure 1. In this example, the same four values ​​used in the above example (depicted in row 405) are converted to a hierarchical shared exponential floating-point data type that stores shared exponential values ​​using a hierarchy of shared exponential values ​​having two levels. The example begins with the HASEFPDT manager 105 retrieving the definitions for the hierarchical shared exponential floating-point data type from the floating-point data storage 125. The HASEFPDT manager 105 then retrieves the values ​​in row 405 from the numeric data storage 120 and converts them to floating-point values ​​(e.g., half-precision floating-point values, single-precision floating-point values, double-precision floating-point values, etc.). Next, the HASEFPDT manager 105 sends a request to the exponent manager 110 to determine the floating-point value, the definition for the hierarchical shared exponent floating-point data type, and the exponent value for the floating-point value.

[0031]

[0039] After receiving definitions and requests for floating-point values ​​and hierarchical shared exponential floating-point data types, the exponential manager 110 determines the exponential value for each floating-point value, as shown in line 410. In this example, the exponential manager 110 determines the shared exponential value for each pair of original exponential values ​​as the shared exponential value for level 0. In this example, the exponential manager 110 determines the shared exponential value for level 0 by determining the exponential value that has the highest value for each pair of original exponential values ​​and using it as the shared exponential value for level 0. As shown in line 415, the highest exponential value among the exponential values ​​for the first pair of values ​​is -3, and the highest exponential value among the exponential values ​​for the second pair of values ​​is -2. The exponential manager 110 then determines the shared exponential for level 1 based on the shared exponential value for level 0. In this example, the exponential manager 110 determines the shared exponential for level 1 by determining the shared exponential value for level 0 that has the highest value and using it as the shared exponential value for level 1. Here, the exponent manager 110 determines that the shared exponent value for level 1 is -2, as shown in line 420. Next, the exponent manager 110 determines the scale value for level 1 for each pair of shared exponent values ​​for level 0 and the corresponding shared exponent values ​​for level 1. In this example, the exponent manager 110 determines the scale value for level 1 by determining the difference between the shared exponent value for level 0 and the corresponding shared exponent value for level 1. If the difference is greater than 1, the exponent manager 110 approximates the scale value for level 1 to 1.

[0032]

[0040] For a first pair of Level 0 shared exponential values ​​and corresponding Level 1 shared exponential values, the exponential manager 110 determines that the Level 1 scale value is 1 (i.e., the difference between -2 and -3 is 1). For a second pair of Level 0 shared exponential values ​​and corresponding Level 1 shared exponential values, the exponential manager 110 determines that the Level 1 scale value is 0 (i.e., the difference between -2 and -2 is 0). The Level 1 scale value is shown in row 425. Row 430 depicts the total scale value, which is the same as the Level 1 scale value. Row 435 shows the total shared exponential scale value for each floating-point value, which is the exponential value determined for the floating-point value by subtracting the respective total scale value from the Level 1 shared exponential value. Row 440 depicts the difference between the determined exponential value and the original exponential value for each floating-point value. Row 445 shows the quantized magnitude of the mantissa for the floating-point value. In this example, a 3-bit mantissa is used, with the first bit used for the sign (not shown in Table 400) and the second bit used for the magnitude. Row 450 shows the quantized exponential value for each floating-point value, which is the same as the shared exponential scale value in row 435. Row 455 then depicts the decimal representation of the quantized floating-point value after converting the original floating-point value to a hierarchical shared exponential floating-point data type.

[0033]

[0041] Once the Level 1 shared exponent and Level 1 scale value are determined, the exponent manager 110 sends them to the HASEFPDT manager 105. The HASEFPDT manager 105 sends a request to the mantissa manager 115 to determine the floating-point value in line 405, the Level 1 shared exponent, the Level 1 scale value, the definition for the hierarchical shared exponent floating-point data type, and the mantissa value for the floating-point value. Upon receiving the data and request, the mantissa manager 115 determines the mantissa value for each floating-point value. If the exponent difference (shown in line 440) is greater than 1, the mantissa manager 115 quantizes the magnitude of the mantissa for each floating-point value to 0. Otherwise, the mantissa manager 115 uses the available bits (1 bit for sign and 1 bit for magnitude in this example) and the respective quantized exponent value (shown in line 450) to best approximate the original floating-point value. The mantissa manager 115 determines the quantized magnitude value for each floating-point value to the value shown in line 445. The mantissa manager 115 then sends the determined mantissa values ​​to the HASEFPDT manager 105.

[0034]

[0042] When the HASEFPDT manager 105 receives a mantissa from the mantissa manager 115, the HASEFPDT manager 105 stores the mantissa, the Level 1 shared exponent, and the Level 1 scale value in a data structure configured to store data for a hierarchical shared exponent floating-point data type. Figure 5 shows an exemplary data structure 500 for storing the hierarchical shared exponent floating-point data type shown in Figure 4, according to several embodiments. The data structure 500 can be generated based on a definition for the hierarchical shared exponent floating-point data type. In this example, the definition for the data type specifies four 2-bit mantissa values ​​for four floating-point values, two 2-bit Level 1 scale values, and an 8-bit Level 1 shared exponent value. As depicted, the data structure 500 includes three attributes: a mantissa attribute, a Level 1 scale attribute, and a shared exponent attribute. The mantissa attribute stores the 2-bit mantissa value determined for each floating-point value. The Level 1 scale attribute stores the 2-bit Level 1 scale value determined for each floating-point value. The shared exponent attribute value stores the 8-bit Level 1 shared exponent value determined for the floating-point value. In this example, the Level 1 shared exponent value is represented using a bias of 127. Therefore, in this example, the value 125 (i.e., 127 + (-2)) is stored in the shared exponent attribute.

[0035]

[0043] Referring to Figures 1-5, the examples described above illustrate how floating-point numbers can be converted to two different hierarchical shared exponential floating-point data types. Those skilled in the art will understand that additional and / or different hierarchical shared exponential floating-point data types can be used in some embodiments. For example, a hierarchical shared exponential floating-point data type may represent a different number of floating-point values ​​that share an exponent (each of the above examples represents four floating-point values), use a different number of bits to represent the mantissa, level scale value and shared exponential value, or use a different number of levels in the exponential value hierarchy (e.g., a different number of levels for the scale value).

[0036]

[0044] In addition, the examples described above with reference to Figures 2-5 illustrate how the hierarchy of shared exponential values ​​can be represented by the highest level shared exponential and level scale values. For example, the example shown in Figure 2 uses a hierarchy of shared exponentials with three different levels: a level 0 exponential value, a level 1 shared exponential value, and a level 2 shared exponential value. As illustrated by Figure 3, the hierarchy of shared exponential values ​​is represented by the level 2 shared exponential value, the level 2 scale value, and the level 1 scale value. The quantized exponential value of the floating-point value (the value in row 260) can be determined from the level 2 shared exponential value, the level 2 scale value, and the level 1 scale value. As another example, the example shown in Figure 4 uses a hierarchy of shared exponentials with two different levels: a level 0 exponential value and a level 1 shared exponential value. As illustrated by Figure 5, the hierarchy of shared exponential values ​​is represented by the level 1 shared exponential value and the level 1 scale value. The quantized exponential value of the floating-point value (the value in row 450) can be determined from the level 1 shared exponential value and the level 1 scale value. When the HASEFPDT manager 105 reads floating-point values ​​stored according to a hierarchical shared exponential floating-point data type (for example, determining quantized values ​​as shown in rows 265 and 455), it determines the quantized exponential value in this manner.

[0037]

[0045] Furthermore, the examples described above, with reference to Figures 2-5, determine the shared index value by selecting the highest value among the lower-level indices / shared indices. Those skilled in the art will understand that additional and / or different statistical measures can be used to determine the shared index value at a particular level. Examples of such measures include weighted average, mean + a defined number of standard deviations (e.g., 1 standard deviation, 2 standard deviations, 3 standard deviations, etc.).

[0038] Exemplary uses of hierarchical shared exponential floating-point data types

[0046] The examples described above illustrate how to create hierarchical shared exponential floating-point data types. The following examples illustrate one of the many uses of these data types. Figure 6 shows an example of multiplying hierarchical shared exponential floating-point data type variables by several embodiments. Specifically, Figure 6 shows a hardware architecture 600 for performing a dot product between two variables stored according to a hierarchical shared exponential floating-point data type. In some embodiments, the hardware architecture 600 can be used to implement an artificial intelligence (AI) accelerator used for training neural networks and inference using neural networks. As shown, the hardware architecture 600 includes multipliers 605-1 to 605-n, shifters 610-1 to 610-n and 620-1 to 620-m, adders 615-1 to 615-m, 625-1 to 625-k and 630, and a truncation / rounding module 635.

[0039]

[0047] In this example, the hardware architecture 600 is configured to receive two inputs A and B. Each of inputs A and B is a hierarchical shared exponential floating-point data type variable (e.g., data structure 300, data structure 500, etc.) that stores n floating-point values. The mantissa of the floating-point values ​​of each variable is input to the corresponding multiplier 605. For example, the mantissa of the first floating-point value of variable A and the mantissa of the first floating-point value of variable B are input to multiplier 605-1, the mantissa of the second floating-point value of variable A and the mantissa of the second floating-point value of variable B are input to multiplier 605-2, the mantissa of the third floating-point value of variable A and the mantissa of the third floating-point value of variable B are input to multiplier 605-3, and so on. As shown, each input to multiplier 605 has a bit width of dw. Each multiplier 605 multiplies two input mantissas and outputs a product with a bit width of 2dw. The output of the multiplier 605 is fed into the corresponding shifter 610, which is configured to perform a right shift based on a control input drawn on top of the shifter 610. The control input for the shifter 610 is the sum of the level 1 scale values ​​of each input to the multiplier 605. The shifter 610 performs a number of right shift operations equal to the value of the control input. The output of the shifter 610 has a bit width of 2dw+3. The outputs of the two shifters 610 are fed into the corresponding adder 615, which sums the two inputs to produce a sum with a bit width of 2dw+4. The process continues to use the corresponding level scale values ​​to control the shifters (for example, the sum of the level 2 scale values ​​is used to control shifters 620-1 to 620-m). The number of levels is determined by log2(n). As mentioned above, n is the number of floating-point values ​​stored in the variable. The output of the last level has a bit width of 2dw + 4log2(n).

[0040]

[0048] When the truncation / rounding module 635 receives output from the last level, it performs a shift operation on the output value based on the sum of the shared exponents of each variable. For example, if the sum of the shared exponents is positive, the truncation / rounding module 635 performs a left shift operation on the value equal to the sum of the shared exponents. If the sum of the shared exponents is negative, the truncation / rounding module 635 performs a right shift operation on the value equal to the absolute value of the sum of the shared exponents. The truncation / rounding module 635 then truncates the value to a defined number of bits to produce the output value C.

[0041]

[0049] Figure 7 shows a process 700 for creating hierarchical shared exponential floating-point data type variables according to several embodiments. In some embodiments, a computing system 100 performs the process 700. The process 700 begins in 710 by determining a first shared exponential value based on the exponential value of each floating-point value in a first subset of multiple floating-point values. Referring to Figures 1 and 4 as an example, the exponential manager 110 may determine a level 0 shared exponential value based on the original exponential values ​​of a first pair of floating-point values.

[0042]

[0050] Next, process 700 determines a second shared exponent value in 720 based on the exponent values ​​of each floating-point value in a second subset of multiple floating-point values. Referring to Figures 1 and 4 as an example, the exponent manager 110 can determine a shared exponent value of level 0 based on the original exponent values ​​of a second pair of floating-point values.

[0043]

[0051] Next, process 700 determines a third shared index value in 730 based on the first and second shared index values. Referring to Figures 1 and 4 as an example, the index manager 110 can determine a level 1 shared index value based on a level 0 shared index value.

[0044]

[0052] In step 740, process 700 determines a first difference value based on a first shared exponential value and a third shared exponential value. Referring to Figures 1 and 4 as an example, the exponential manager 110 can determine a Level 1 scale value for a first pair of Level 0 shared exponential values ​​and corresponding Level 1 shared exponential values. Then, in step 750, process 700 determines a second difference value based on a second shared exponential value and a third shared exponential value. Referring to Figures 1 and 4 as an example, the exponential manager 110 can determine a Level 1 scale value for a second pair of Level 0 shared exponential values ​​and corresponding Level 1 shared exponential values.

[0045]

[0053] Next, process 700 determines the sign and mantissa values ​​for each of the multiple floating-point values ​​in 760. Referring to Figures 1 and 4 as an example, the mantissa manager 115 can determine the quantized magnitude value for the mantissa, as shown in row 445 of Table 400. Finally, in 770, process 700 stores the sign and mantissa values ​​for each of the multiple floating-point values, the third shared exponent value, the first difference value, and the second difference value in a data structure for the shared exponent floating-point data type. Referring to Figures 1 and 5 as an example, the HASEFPDT manager 105 stores the mantissa value, the Level 1 shared exponent, and the Level 1 scale value in the data structure 500.

[0046]

[0054] The techniques described above can be implemented in a wide range of computer systems configured to process neural networks. Figure 8 depicts a simplified block diagram of an exemplary computer system 800 (e.g., computing system 100) that can be used to implement the techniques described in the preceding disclosure. As shown in Figure 8, computer system 800 includes one or more processors 802 that communicate with many peripheral devices via a bus subsystem 804. These peripheral devices may include a storage subsystem 806 (e.g., including a memory subsystem 808 and a file storage subsystem 810) and a network interface subsystem 816. Some computer systems may further include user interface input devices 812 and / or user interface output devices 814.

[0047]

[0055] The bus subsystem 804 can provide a mechanism for various components and subsystems of the computer system 800 to communicate with each other as intended. Although the bus subsystem 804 is schematically shown as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses.

[0048]

[0056] The network interface subsystem 816 can serve as an interface for communicating data between the computer system 800 and other computer systems or networks. Embodiments of the network interface subsystem 816 may include, for example, Ethernet, Wi-Fi and / or cellular adapters, modems (telephone, satellite, cable, ISDN, etc.), digital subscriber line (DSL) units and / or similar.

[0049]

[0057] The storage subsystem 806 includes a memory subsystem 808 and a file / disk storage subsystem 810. The subsystems 808, 810 and other memories described herein are examples of non-temporary computer-readable storage media capable of storing executable program code and / or data that provide the functionality of embodiments of the present disclosure.

[0050]

[0058] The memory subsystem 808 includes a large amount of memory, including a main random access memory (RAM) 818 for storing instructions and data during program execution, and a read-only memory (ROM) 820 for storing fixed instructions. The file storage subsystem 810 can provide persistent (e.g., non-volatile) storage for program and data files and may include magnetic or solid-state hard disk drives, optical drives with associated removable media (e.g., CD-ROM, DVD, Blu-ray, etc.), removable flash memory-based drives or cards, and / or other types of storage media known in the art.

[0051]

[0059] Computer system 800 is illustrative, and it should be understood that many other configurations with more or fewer components than system 800 are possible.

[0052]

[0060] Figure 9 shows a neural network processing system according to several embodiments. In various embodiments, the neural network according to this disclosure can be implemented and trained in a hardware environment including one or more neural network processors. The neural network processor may refer to, for example, various graphics processing units (GPUs) (e.g., GPUs for processing neural networks produced by Nvidia Corp®), field-programmable gate arrays (FPGAs) (e.g., FPGAs for processing neural networks produced by Xilinx®), or various application-specific integrated circuits (ASICs) or neural network processors including hardware architectures optimized for neural network computation. In this exemplary environment, one or more servers 902, which may include the architecture shown in Figure 8 above, can be coupled to a plurality of controllers 910(1) to 910(M) on a communication network 901 (e.g., switches, routers, etc.). The controllers 910(1) to 910(M) may also include the architecture shown in Figure 8 above. Each controller 910(1) to 910(M) can be coupled to one or more NN processors (e.g., processors 911(1) to 911(N) and 912(1) to 912(N)). The NN processors 911(1) to 911(N) and 912(1) to 912(N) may include functional processing blocks and memory in various configurations optimized for neural network processing (e.g., training or inference). The NN processors are optimized for neural network computation. The server 902 can use NN models and configure the controllers 910 to input data into the models, and these models can be loaded and executed in parallel by, for example, the NN processors 911(1) to 911(N) and 912(1) to 912(N). The models may include, for example, layers and associated weights as described above. The NN processors can load the models, apply the inputs, and produce output results. The NN processors can also implement, for example, the training algorithms described herein.

[0053] Further exemplary embodiments

[0061] In various embodiments, the Disclosure includes systems, methods, and apparatus for providing hierarchical shared exponential floating-point data types. The techniques described herein can be embodied in a non-temporary machine-readable medium that stores a program executable by a computer system, the program including a set of instructions for performing the techniques described herein. In some embodiments, the system includes a set of processing units and a non-temporary machine-readable medium that stores instructions, the instructions, when executed by at least one processing unit of the set of processing units, cause at least one processing unit to perform the techniques described herein. In some embodiments, the non-temporary machine-readable medium may be, for example, memory, and may be coupled, for example, to one or more controllers or one or more artificial intelligence processors.

[0054]

[0062] The following techniques can be embodied individually or in different combinations, and can be further embodied by other techniques described herein.

[0055]

[0063] For example, in one embodiment, the disclosure includes a non-temporary machine-readable medium for storing a program executable by at least one processing unit of a device. The program includes a set of instructions for determining a first shared exponent value based on the exponent values ​​of each floating-point value in a first subset of multiple floating-point values; determining a second shared exponent value based on the exponent values ​​of each floating-point value in a second subset of multiple floating-point values; determining a third shared exponent value based on the first and second shared exponents; determining a first difference value based on the first and third shared exponents; determining a second difference value based on the second and third shared exponents; determining a sign and mantissa value for each floating-point value in the multiple floating-point values; and storing the sign and mantissa values ​​for each floating-point value in the multiple floating-point values, the third shared exponent value, the first difference value, and the second difference value in a data structure for a shared exponent floating-point data type.

[0056]

[0064] In one embodiment, the first shared index value, the second shared index value, and the third shared index value form a hierarchy of shared index values.

[0057]

[0065] In one embodiment, the first shared index value and the second shared index value are index values ​​at the first level of the shared index value hierarchy, and the third shared index value is an index value at the second level of the shared index value hierarchy.

[0058]

[0066] In one embodiment, the hierarchy of shared index values ​​is represented by a third shared index value, a first difference value, and a second difference value.

[0059]

[0067] In one embodiment, the disclosure further determines a first set of difference values ​​based on a first shared exponential value and the exponential values ​​of each floating-point number in a first subset of a plurality of floating-point values; determines a second set of difference values ​​based on a second shared exponential value and the exponential values ​​of each floating-point number in a second subset of a plurality of floating-point values; and stores the first set of difference values ​​and the second set of difference values ​​in a data structure for a shared exponential floating-point data type.

[0060]

[0068] In one embodiment, determining a first shared exponent value includes determining the exponent value of a floating-point value of a first subset of multiple floating-point values ​​having the highest value, and using that exponent value as the first shared exponent value.

[0061]

[0069] In one embodiment, determining a second shared exponent value includes determining the exponent value of a second subset of floating-point values ​​having the highest value, and using that exponent value as the second shared exponent value.

[0062]

[0070] In one embodiment, determining a third shared index value includes using the first shared index value as the third index value if it is determined that the first shared index value is greater than or equal to the second shared index value, and using the second shared index value as the third index value if it is determined that the first shared index value is less than the second shared index value.

[0063]

[0071] In one embodiment, the number of bits used to represent the dummy value stored in the data structure for the shared exponential floating-point data type is less than the number of bits used to represent the third exponential value.

[0064]

[0072] The above description illustrates various embodiments of the Disclosure, along with examples showing how aspects of a particular embodiment can be implemented. These examples should not be considered sole embodiments, but are presented to demonstrate the flexibility and advantages of a particular embodiment, as defined by the following claims. Based on the above Disclosure and the following claims, other arrangements, embodiments, implementations, and equivalents can be adopted without departing from the scope of the Disclosure as defined by the claims.

Claims

1. A non-temporary machine-readable medium for storing a program executable by at least one processing unit of a device, wherein the program is: Determining a first shared exponent value based on the exponent values ​​of each floating-point value in a first subset of multiple floating-point values, A second shared exponent is determined based on the exponent of each floating-point value in a second subset of the aforementioned plurality of floating-point values, Based on the first shared index value and the second shared index value, a third shared index value is determined. Based on the first shared index value and the third shared index value, the first difference value is determined, Based on the second shared index value and the third shared index value, the second difference value is determined, Determining the sign value and mantissa value for each of the aforementioned plurality of floating-point values, The sign value and the mantissa value for each of the plurality of floating-point values, the third shared exponent value, the first difference value, and the second difference value are stored in a data structure for a shared exponent floating-point data type. The present invention provides the data structure as input to a set of hardware components, the set of hardware components configured to perform a set of operations on the plurality of floating-point values ​​based on the data structure for the shared exponential floating-point data type, and to generate an output based on the set of operations, the set of hardware components including a set of multipliers for performing multiplication operations on the mantissa values ​​of the plurality of floating-point values ​​to generate a plurality of products, a first set of shifters performing a shift operation on the plurality of products based on the first difference value and the second difference value to generate a first plurality of shifted values, a first set of adders adding the first plurality of shifted values ​​to generate a plurality of sums, a second set of shifters performing a shift operation on the plurality of sums to generate a second plurality of shifted values, and a second set of adders adding the second plurality of shifted values ​​to generate a set of sums. A non-temporary, machine-readable medium containing a set of instructions for performing a certain action.

2. The non-temporary machine-readable medium according to claim 1, wherein the first shared index value, the second shared index value, and the third shared index value form a hierarchy of shared index values.

3. The non-temporary machine-readable medium according to claim 2, wherein the first shared index value and the second shared index value are index values ​​of the first level of the hierarchy of shared index values, and the third shared index value is an index value of the second level of the hierarchy of shared index values.

4. The non-temporary machine-readable medium according to claim 2, wherein the hierarchy of shared index values ​​is represented by the third shared index value, the first difference value, and the second difference value.

5. The aforementioned program, Based on the first shared exponential value and the exponential values ​​of each floating-point number in the first subset of the plurality of floating-point values, a first set of difference values ​​is determined. Based on the second shared exponential value and the exponential values ​​of each floating-point number in the second subset of the plurality of floating-point values, a second set of difference values ​​is determined. The first set of different values ​​and the second set of the difference values ​​are stored in the data structure for the shared exponential floating-point data type. A non-temporary machine-readable medium according to claim 1, further comprising a set of instructions for performing the following.

6. The non-temporary machine-readable medium according to claim 1, wherein determining the first shared exponent value comprises determining the exponent value of a first subset of floating-point values ​​having the highest value, and using the exponent value as the first shared exponent value.

7. The non-temporary machine-readable medium according to claim 1, wherein determining the second shared exponent value comprises determining the exponent value of a second subset of floating-point values ​​having the highest value, and using the exponent value as the second shared exponent value.

8. The non-temporary machine-readable medium according to claim 1, wherein determining the third shared index value includes using the first shared index value as the third shared index value if it is determined that the first shared index value is greater than or equal to the second shared index value, and using the second shared index value as the third shared index value if it is determined that the first shared index value is less than the second shared index value.

9. The non-temporary machine-readable medium according to claim 1, wherein the number of bits used to represent the dummy value stored in the data structure for the shared exponential floating-point data type is less than the number of bits used to represent the third shared exponential value.

10. Determining a first shared exponent value based on the exponent values ​​of each floating-point value in a first subset of multiple floating-point values, A second shared exponent is determined based on the exponent of each floating-point value in a second subset of the aforementioned plurality of floating-point values, Based on the first shared index value and the second shared index value, a third shared index value is determined. Based on the first shared index value and the third shared index value, the first difference value is determined, Based on the second shared index value and the third shared index value, the second difference value is determined, Determining the sign value and mantissa value for each of the aforementioned plurality of floating-point values, The sign value and the mantissa value for each of the plurality of floating-point values, the third shared exponent value, the first difference value, and the second difference value are stored in a data structure for a shared exponent floating-point data type. The present invention provides the data structure as input to a set of hardware components, the set of hardware components configured to perform a set of operations on the plurality of floating-point values ​​based on the data structure for the shared exponential floating-point data type, and to generate an output based on the set of operations, the set of hardware components including a set of multipliers for performing multiplication operations on the mantissa values ​​of the plurality of floating-point values ​​to generate a plurality of products, a first set of shifters performing a shift operation on the plurality of products based on the first difference value and the second difference value to generate a first plurality of shifted values, a first set of adders adding the first plurality of shifted values ​​to generate a plurality of sums, a second set of shifters performing a shift operation on the plurality of sums to generate a second plurality of shifted values, and a second set of adders adding the second plurality of shifted values ​​to generate a set of sums. A method that includes this.

11. The method according to claim 10, wherein the first shared index value, the second shared index value, and the third shared index value form a hierarchy of shared index values.

12. The method according to claim 11, wherein the first shared index value and the second shared index value are index values ​​of the first level in the hierarchy of shared index values, and the third shared index value is an index value of the second level in the hierarchy of shared index values.

13. The method according to claim 11, wherein the hierarchy of the shared index values ​​is represented by the third shared index value, the first difference value, and the second difference value.

14. Based on the first shared exponential value and the exponential values ​​of each floating-point number in the first subset of the plurality of floating-point values, a first set of difference values ​​is determined. Based on the second shared exponential value and the exponential values ​​of each floating-point number in the second subset of the plurality of floating-point values, a second set of difference values ​​is determined. The first set of different values ​​and the second set of the difference values ​​are stored in the data structure for the shared exponential floating-point data type. The method according to claim 10, further comprising:

15. The method according to claim 10, wherein determining the first shared exponent value includes determining the exponent value of a first subset of floating-point values ​​having the highest value, and using the exponent value as the first shared exponent value.

16. The method according to claim 10, wherein determining the second shared exponent value includes determining the exponent value of a second subset of floating-point values ​​having the highest value, and using the exponent value as the second shared exponent value.

17. The method according to claim 10, wherein determining the third shared index value includes using the first shared index value as the third shared index value if it is determined that the first shared index value is greater than or equal to the second shared index value, and using the second shared index value as the third shared index value if it is determined that the first shared index value is less than the second shared index value.

18. The method according to claim 10, wherein the number of bits used to represent the dummy value stored in the data structure for the shared exponential floating-point data type is less than the number of bits used to represent the third shared exponential value.

19. A set of processing units, A non-temporary machine-readable medium for storing instructions, A system including, When the instruction is executed by at least one processing unit of the set of processing units, Determining a first shared exponent value based on the exponent values ​​of each floating-point value in a first subset of multiple floating-point values, A second shared exponent is determined based on the exponent of each floating-point value in a second subset of the aforementioned plurality of floating-point values, Based on the first shared index value and the second shared index value, a third shared index value is determined. Based on the first shared index value and the third shared index value, the first difference value is determined, Based on the second shared index value and the third shared index value, the second difference value is determined, Determining the sign value and mantissa value for each of the aforementioned plurality of floating-point values, The sign value and the mantissa value for each of the plurality of floating-point values, the third shared exponent value, the first difference value, and the second difference value are stored in a data structure for a shared exponent floating-point data type. The present invention provides the data structure as input to a set of hardware components, the set of hardware components configured to perform a set of operations on the plurality of floating-point values ​​based on the data structure for the shared exponential floating-point data type, and to generate an output based on the set of operations, the set of hardware components including a set of multipliers for performing multiplication operations on the mantissa values ​​of the plurality of floating-point values ​​to generate a plurality of products, a first set of shifters performing a shift operation on the plurality of products based on the first difference value and the second difference value to generate a first plurality of shifted values, a first set of adders adding the first plurality of shifted values ​​to generate a plurality of sums, a second set of shifters performing a shift operation on the plurality of sums to generate a second plurality of shifted values, and a second set of adders adding the second plurality of shifted values ​​to generate a set of sums. A system that causes at least one processing unit to perform the above.

20. The aforementioned instruction is, Based on the first shared exponential value and the exponential values ​​of each floating-point number in the first subset of the plurality of floating-point values, a first set of difference values ​​is determined. Based on the second shared exponential value and the exponential values ​​of each floating-point number in the second subset of the plurality of floating-point values, a second set of difference values ​​is determined. Storing a first set of different values ​​and a second set of the difference values ​​in the data structure for the shared exponential floating-point data type, The system according to claim 19, wherein the processing is performed by at least one processing unit.

Images