Hierarchical Shared-Exponent Floating-Point Data Type

Abstract

Embodiments of the present disclosure include systems and methods for providing a hierarchical shared-exponent floating-point data type. First and second shared exponent values ​​are determined based on the exponent values ​​of the plurality of floating-point values. A third shared exponent value is determined based on the first shared exponent value and the second shared exponent value. First and second difference values ​​are determined based on the first shared exponent value, the second shared exponent value and the third shared exponent value. A sign value and a mantissa value are determined for the plurality of floating-point values. The sign value and mantissa value, the third shared exponent value, the first difference value and the second difference value for each floating-point value of the plurality of floating-point values ​​are stored in a data structure for the shared-exponent floating-point data type.

Description

[Technical field]

[0001] background

[0001] The present disclosure relates to computing hardware. More specifically, the present disclosure relates to floating-point data types. [Background technology]

[0002]

[0002] Neural networks are machine learning models used for a variety of different applications (e.g., image classification, computer vision, natural language processing, speech recognition, handwriting recognition, etc.) Neural networks can be trained for a specific purpose by running a data set through the neural network, comparing the results from the neural network with known results, and updating the network based on the differences.

[0003]

[0003] Efficient training of neural networks and inference using neural networks on low-fidelity data types may require developing data types that maximize the fidelity of each bit while minimizing the computing cost. This can be formulated as an optimization problem, where the objective is to maximize the quantized signal-to-noise ratio (QSNR) metric while minimizing the area overhead of a hardware dot-product unit. Summary of the Invention [Means for solving the problem]

[0004] BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Brief description of the drawings]

[0005] [Figure 1] 1 illustrates a computing system according to some embodiments. [Diagram 2]

[0006] 1 illustrates an example of converting a floating-point value to a hierarchical shared-exponent floating-point data type according to some embodiments. [Diagram 3]

[0007] 3 illustrates an example data structure for storing the hierarchical shared-exponent floating-point data type shown in FIG. 2 in accordance with some embodiments. [Figure 4]

[0008] 1 illustrates another example of converting a floating-point value to a hierarchical shared-exponent floating-point data type according to some embodiments. [Diagram 5]

[0009] 5 illustrates an example data structure for storing the hierarchical shared-exponent floating-point data type shown in FIG. 4 according to some embodiments. [Figure 6]

[0010] 1 illustrates an example of multiplying hierarchical shared exponent floating-point data type variables according to some embodiments. [Figure 7]

[0011] 1 illustrates a process for creating hierarchical shared exponent floating point data type variables according to some embodiments. [Figure 8]

[0012] 1 depicts a simplified block diagram of an exemplary computer system according to some embodiments. [Figure 9]

[0013] 1 illustrates a neural network processing system according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Detailed Description

[0014] In the following description, for the purpose of explanation, numerous examples and specific details are set forth to provide a detailed understanding of the present disclosure. Such examples and details should not be construed as unduly limiting the elements of the claims or the claimed subject matter as a whole. It will be apparent to those skilled in the art that, based on the language of different claims, the claimed subject matter may include some or all of the features of these examples, alone or in combination, and may further include modifications and equivalents of the features and techniques described herein.

[0007]

[0015] A technique for providing a hierarchical shared-exponent floating-point data type is now described. In some embodiments, a computing system can be configured to create and manage data of a hierarchical shared-exponent floating-point data type. For example, the computing system can create data of a hierarchical shared-exponent floating-point data type from several numerical values ​​that are converted to floating-point numbers (e.g., half-precision floating-point numbers, single-precision floating-point numbers, double-precision floating-point numbers, etc.) as needed. To achieve this, the computing system first determines an exponent value for each of the floating-point numbers. The computing system then quantizes these exponent values ​​based on a hierarchy of shared exponent values. The computing system then quantizes a mantissa of each of the floating-point numbers based on the quantized exponent values. Finally, the computing system stores the hierarchy of shared exponent values ​​and the quantized mantissa in a data structure for a hierarchical shared-exponent floating-point data type.

[0008]

[0016] In some embodiments, efficient training / inference of ultra-narrow bit-widths (e.g., 1-4 bit mantissas) may require developing a data type that maximizes the fidelity of each bit while minimizing the computing cost. This can be formulated as an optimization problem, where the objective is to maximize the quantized signal-to-noise ratio (QSNR) while minimizing the area overhead of the hardware dot-product unit. As discussed, the hierarchical shared-exponential floating-point data type described herein can further push the Pareto frontier of QSNR vs. multiply-accumulate (MAC) area and unlock further efficiencies compared to other floating-point data types. In this ultra-narrow bit-width regime, such an increase in effective QSNR can directly translate to better achievable accuracy for both training neural networks and inference using those neural networks.

[0009]

[0017] In some embodiments, the data type described herein is a shared exponent data type. That is, the exponent is shared among a small number of elements. The number of elements can be referred to as the tile size or block size. Using a small tile size may not be the best choice from a hardware perspective, because with the reduction in the number of mantissa bits, most of the cost of the integer multiplier is eliminated and the accumulator starts to take up the total area of ​​the multiplier-accumulator (MAC) unit. Thus, the adoption of a relatively large tile size can amortize the accumulator silicon cost while avoiding excessive mapping of elements within the tile to zero due to the difference between their private exponent and the shared exponent of the tile.

[0010]

[0018] In some embodiments, the data types described herein aim to provide a balance between allowable tile size and coding efficiency in the ultra-narrow bitwidth regime, which can be defined by the following equation:

[0011]

[0019]

number

[0012]

[0020] During the ceremony,

number

[0013]

[0021] In some embodiments, a hierarchical shared-exponent floating-point data type decomposes each tile into nested subtiles, where the shared exponent of each subtile may differ from the subtile at the next level by a scale value. These scales may be implemented as right shifts of an adder tree unfolded in a dot-product unit.

[0014]

[0022] In some embodiments, the data types described herein have several adjustable parameters. One parameter is the tile size, which is the number of elements that share a global exponent. Another parameter is the number of mantissa bits. In some embodiments, the mantissa values ​​are represented using two's complement numbers, and in other embodiments, the mantissa values ​​are represented using a sign / magnitude format. The number of mantissa bits can take any value equal to or greater than 1.

[0015]

[0023] Yet another parameter is the representation of the shared exponent. In some embodiments, the shared exponent can be represented using a single level. For a single level representation, one parameter is the subtile size, which is the number of elements that share a local scale to adjust the global exponent. Each tile can be divided into multiple subtiles of equal size m, each subtile sharing a common subscale. If m=1, a right shifter can be added to the leaves of the adder tree in the dot product unit. The larger the value of m, the closer the right shifter is to the root of the adder tree. For example, if m=16, the right shifter is added to the Log2(l6)=4th level of the adder tree. Another parameter of the single level representation is the subtile scale, which specifies the amount of right shift allowed per element. The scale encodes the maximum allowed difference between the global shared exponent and the exponent selected within the subtile. For example, a scale of 1 at the leaves of the adder tree means that each input element can be shifted by 0 or 1, resulting in multiplexing of 2-bit combinations. In some cases, the maximum exponent per tile can be used as the global shared exponent. Thus, the scale is always a positive number greater than or equal to zero. In some embodiments, other exponent selection techniques can be used. In instances where the global shared exponent is not the maximum exponent of the tile, the selected scale can be a positive or negative value.

[0016]

[0024] In another embodiment, the shared exponent is represented hierarchically. Instead of using a large single-level shift to scale the shared exponent, the shift is distributed over multiple levels of the adder tree, and the scale per shift is constrained instead. One parameter of the hierarchical representation is the hierarchical level, which is the number of hierarchical levels of the adder tree used to adjust the shared exponent. The indexing starts from the leaves of the tree. For example, with a tile size of 32, there will be a maximum of 5 levels in the adder tree. A hierarchical level of 3 implies the use of right shifts within the first 3 levels starting from the leaves of the adder tree. Another parameter of the hierarchical representation is the subtile scale, which is the maximum scale per element allowed at each level. The scale encodes the difference with the shared exponent of the next level (not necessarily the global exponent). The shared exponent may or may not be the maximum exponent. As an example, if the maximum and minimum exponents of a tile are computed, the difference between the maximum and the minimum gives the maximum scale difference. If the hierarchical level is set to 5, then this exponent difference (max-min) can be capped at 5 (because there are 5 levels of subtiles) and the global exponent can be set to (max-min).clip(5) / 2. This causes the tile to be split into subtiles hierarchically, and at each level it is checked whether or not a -1 / 0 / 1 scale should be set for each subtile based on the subtile's exponent.

[0017]

[0025] The accumulator bit width parameter specifies the data type used for accumulation at the end of the adder tree. Examples of such data types include half-precision floating-point data types, single-precision floating-point data types, double-precision floating-point data types, etc. In some embodiments, the default data type may be single-precision floating-point values. The truncation bit width may be another parameter. This parameter specifies the amount of truncation after each integer addition, which may effectively reduce area overhead. An example of an effective truncation scheme is to truncate the output of integer multipliers at the leaves of the adder tree with an 8-bit mantissa, such that the bit width of the output of integer adders at each level of the adder tree increases by one. Such an approach may, for example, reduce area overhead while minimally affecting the achievable QSNR (i.e., accuracy).

[0018]

[0026] 1 illustrates a computing system 100 according to some embodiments. As shown, the computing system 100 includes a hierarchical shared exponent floating-point data type (HASEFPDT) manager 105, an exponent manager 110, a mantissa manager 115, a numeric data storage 120, and a floating-point data storage 125. The numeric data storage 120 stores numeric data that can be converted to a hierarchical shared exponent floating-point data type. Examples of numeric data include floating-point numbers, integers, etc. The floating-point data storage 125 is configured to store a hierarchical shared exponent floating-point data type. Examples of such data types are described in more detail below. Additionally, the floating-point data storage 125 can store a definition for a hierarchical shared exponent floating-point data type.

[0019]

[0027] The HASEFPDT manager 105 is responsible for managing the hierarchical shared exponent floating-point data type and data stored according to such data type. For example, the HASEFPDT manager 105 can create data stored as a hierarchical shared exponent floating-point data type by retrieving a definition for the hierarchical shared exponent floating-point data type from the floating-point data storage 125. The HASEFPDT manager 105 can then retrieve numeric data from the numeric data storage 120 and convert the numeric data to a floating-point number (e.g., a half-precision floating-point number, a single-precision floating-point number, a double-precision floating-point number, etc.). The number of numeric data that the HASEFPDT manager 105 retrieves can be based on what is specified in the definition for the hierarchical shared exponent floating-point data type. The HASEFPDT manager 105 then sends a request to the exponent manager 110 to determine the converted numeric data, the definition for the hierarchical shared exponent floating-point data type, and the shared exponent value for the numeric data. When HASEFPDT manager 105 receives the shared exponent value from exponent manager 110, HASEFPDT manager 105 sends a request to mantissa manager 115 to determine the numeric data, the shared exponent value, a definition for the hierarchical shared-exponent floating-point data type, and a mantissa value for the numeric data. Once HASEFPDT manager 105 receives the mantissa value from mantissa manager 115, HASEFPDT manager 105 stores the mantissa value and the shared exponent value in a data structure configured to store data for the hierarchical shared-exponent floating-point data type.

[0020]

[0028] The HASEFPDT manager 105 can also read values ​​from data of a hierarchical shared-exponent floating-point data type. For example, the HASEFPDT manager 105 can receive a request (e.g., from a component operating on the computing system 100, from a component operating on another computing system, etc.) to read a particular value from data stored according to a hierarchical shared-exponent floating-point data type (e.g., a data structure for storing data according to a hierarchical shared-exponent floating-point data type). In response, the HASEFPDT manager 105 retrieves a mantissa value for the requested value from the hierarchical shared-exponent floating-point data type data, determines an exponent value for the requested value based on the shared exponent value stored in the hierarchical shared-exponent floating-point data type data, and generates the requested value based on the mantissa value and the exponent value.

[0021]

[0029] Exponent manager 110 is configured to determine exponent values ​​for a hierarchical shared-exponent floating-point data type. For example, exponent manager 110 can receive from HASEFPDT manager 105 a request to determine numeric data, a definition for a hierarchical shared-exponent floating-point data type, and an exponent value for the numeric data. In response, exponent manager 110 determines a hierarchy of exponent values ​​based on the numeric data and the definition for the hierarchical shared-exponent floating-point data type. Exponent manager 110 then transmits the hierarchy of exponent values ​​to HASEFPDT manager 105.

[0022]

[0030] Mantissa manager 115 handles the determination of a mantissa value for a hierarchical shared-exponent floating-point data type. For example, mantissa manager 115 can receive from HASEFPDT manager 105 a request to determine numeric data, a shared exponent value, a definition for a hierarchical shared-exponent floating-point data type, and a mantissa value for the numeric data. Based on the shared exponent value and the definition for the hierarchical shared-exponent floating-point data type, mantissa manager 115 determines a mantissa value for the numeric data. Mantissa manager 115 then sends the mantissa value to HASEFPDT manager 105.

[0023]

[0031] FIG. 2 illustrates an example of converting floating-point values ​​to a hierarchical shared-exponent floating-point data type according to some embodiments. Specifically, FIG. 2 illustrates a table 200 that includes various values ​​throughout the process of converting floating-point values ​​to a hierarchical shared-exponent floating-point data type. This example is described by referring to FIG. 1. For this example, the four values ​​in row 205 are converted to a hierarchical shared-exponent floating-point data type that stores the shared-exponent values ​​using a hierarchy of shared-exponent values ​​having three levels. As shown, the four values ​​are presented as decimal values. The HASEFPDT manager 105 begins by retrieving a definition for the hierarchical shared-exponent floating-point data type from the floating-point data storage 125. Next, the HASEFPDT manager 105 retrieves the values ​​of row 205 from numeric data storage 120 and converts them to floating-point values ​​(e.g., half-precision floating-point values, single-precision floating-point values, double-precision floating-point values, etc.). The HASEFPDT manager 105 then sends a request to the exponent manager 110 to determine the floating-point values, a definition for a hierarchical shared-exponent floating-point data type, and an exponent value for the floating-point values.

[0024]

[0032] Upon receiving the floating-point values, the definition and request for the hierarchical shared exponent floating-point data type, the exponent manager 110 determines the exponent values ​​for each of the floating-point values, as depicted in row 210 of table 200. These exponent values ​​are considered to be level 0 exponent values ​​in this example, as shown in row 215. The exponent manager 110 then determines the level 1 shared exponent value for each pair of exponent values. For this example, the exponent manager 110 determines the level 1 shared exponent value by determining the exponent value with the highest value for each pair of exponent values ​​and using it as the level 1 shared exponent value. As shown in row 220, the highest exponent value among the exponent values ​​for the first pair of values ​​is -3. The highest exponent value among the exponent values ​​for the second pair of values ​​is -2. The exponent manager 110 then determines the level 1 scale value for each pair of level 0 exponent values ​​and the corresponding level 1 shared exponent value. Here, the exponent manager 110 determines the scale value for level 1 by determining the difference between the exponent value for level 0 and the corresponding shared exponent value for level 1. If the difference is greater than 1, then the exponent manager 110 makes the scale value for level 1 approximate 1.

[0025]

[0033] For the first and second pair of level 0 exponent values ​​and corresponding level 1 shared exponent values ​​and the second value, index manager 110 determines that each of the level 1 scale values ​​is 0 (i.e., the difference between -3 and -3 is 0). For the third pair of level 0 exponent values ​​and corresponding level 1 shared exponent values, index manager 110 determines that the difference between the level 0 exponent value and the level 1 shared exponent value is 2. This is greater than 1, so index manager 110 approximates the level 1 scale value to 1. For the fourth pair of level 0 exponent values ​​and corresponding level 1 shared exponent values, index manager 110 determines that the level 1 scale value is 0 (i.e., the difference between -2 and -2 is 0). The level 1 scale values ​​are depicted in row 225 of table 200.

[0026]

[0034] After determining the shared index value and scale value for level 1, index manager 110 determines the shared index value for level 2. In this example, index manager 110 determines the shared index value for level 2 based on the shared index value for level 1. Specifically, index manager 110 determines the shared index value for level 2 by determining the index value that has the highest value among the shared index values ​​for level 1 and using it as the shared index value for level 2. Here, index manager 110 determines that the shared index value for level 2 is −2, as shown in row 230.

[0027]

[0035] Next, index manager 110 determines a level 2 scale value for each pair of level 1 shared exponent values ​​and corresponding level 2 shared exponent values. In this example, index manager 110 determines the level 2 scale value by determining the difference between the level 1 shared exponent value and the corresponding level 2 shared exponent value. If the difference is greater than 1, index manager 110 approximates the level 2 scale value to 1. For a first pair of level 1 shared exponent values ​​and corresponding level 2 shared exponent values, index manager 110 determines that the level 2 scale value is 1 (i.e., the difference between -2 and -3 is 1). For a second pair of level 1 shared exponent values ​​and corresponding level 2 shared exponent values, index manager 110 determines that the level 2 scale value is 0 (i.e., the difference between -2 and -2 is 0). The level 2 scale values ​​are shown in row 235. Row 240 depicts a total scale value for each floating point value, which is the sum of the level 1 scale value and the level 2 scale value. Row 245 depicts a total shared exponent scale value for each floating point value, which is the exponent value determined for the floating point value by subtracting the respective total scale value from the level 2 shared exponent value. Row 250 depicts the difference between the determined exponent value and the original exponent value for each floating point value. Row 255 depicts the quantized magnitude of the mantissa for the floating point values. In this example, a 2-bit mantissa is used, with the first bit used for the sign (not shown in table 200) and the second bit used for the magnitude. Row 260 depicts the quantized exponent value for each of the floating point values, which is the same as the shared exponent scale value in row 245. Finally, line 265 shows the decimal representation of the quantized floating-point value after converting the original floating-point value to a hierarchical shared-exponent floating-point data type.

[0028]

[0036] After the exponent manager 110 determines the level 2 shared exponent, the level 2 scale value, and the level 1 scale value, the exponent manager 110 sends them to the HASEFPDT manager 105. The HASEFPDT manager 105 then sends a request to the mantissa manager 115 to determine the floating-point value of row 205, the level 2 shared exponent, the level 2 scale value, the level 1 scale value, a definition for the hierarchical shared exponent floating-point data type, and a mantissa value for the floating-point value. Once the mantissa manager 115 receives the data and the request, the mantissa manager 115 determines the mantissa value for each floating-point value. If the exponent difference (shown in row 250) is greater than 0, the mantissa manager 115 quantizes the mantissa magnitude for the respective floating-point value to 0. If not, mantissa manager 115 uses the available bits (in this example, one bit for sign and one bit for magnitude) and the respective quantized exponent values ​​(shown on line 260) to best approximate the original floating-point values. Here, mantissa manager 115 determines the quantized magnitude values ​​for each of the floating-point values ​​to the values ​​depicted on line 255. Mantissa manager 115 then sends the determined mantissa values ​​to HASEFPDT manager 105.

[0029]

[0037] Once the HASEFPDT manager 105 receives the mantissa value from the mantissa manager 115, the HASEFPDT manager 105 stores the mantissa value, the level 2 shared exponent, the level 2 scale value, and the level 1 scale value in a data structure configured to store data for a hierarchical shared-exponent floating-point data type. FIG. 3 illustrates an exemplary data structure 300 for storing the hierarchical shared-exponent floating-point data type illustrated in FIG. 2, according to some embodiments. The data structure 300 can be generated based on a definition for the hierarchical shared-exponent floating-point data type, where the definition for the data type specifies four 2-bit mantissa values, four 1-bit level 1 scale values, two 1-bit level 2 scale values, and an 8-bit level 2 shared exponent value for four floating-point values. As illustrated, the data structure 300 includes four attributes: a mantissa attribute, a level 1 scale attribute, a level 2 scale attribute, and a shared exponent attribute. The mantissa attribute stores the 2-bit mantissa value determined for each of the floating-point values. The level 1 scale attribute stores a 1-bit level 1 scale value determined for each of the floating point values. The level 2 scale value attribute stores a 1-bit level 2 scale value determined for each pair of floating point values. The shared exponent attribute value stores an 8-bit level 2 shared exponent value determined for the floating point values. In this example, the value of the level 2 shared exponent value is represented using a bias of 127. Thus, for this example, a value of 125 (i.e., 127 + (-2)) is stored in the shared exponent attribute.

[0030]

[0038] FIG. 4 illustrates another example of converting floating-point values ​​to a hierarchical shared-exponent floating-point data type according to some embodiments. Specifically, FIG. 4 illustrates a table 400 that includes various values ​​throughout the process of converting floating-point values ​​to different hierarchical shared-exponent floating-point data types. This example is also described by referring to FIG. 1. In this example, the same four values ​​(depicted in row 405) used in the above example are converted to a hierarchical shared-exponent floating-point data type that stores the shared exponent values ​​using a hierarchy of shared exponent values ​​having two levels. The example begins with the HASEFPDT manager 105 retrieving a definition for a hierarchical shared-exponent floating-point data type from the floating-point data storage 125. The HASEFPDT manager 105 then retrieves the values ​​in row 405 from the numeric data storage 120 and converts them to floating-point values ​​(e.g., half-precision floating-point values, single-precision floating-point values, double-precision floating-point values, etc.). Next, the HASEFPDT manager 105 sends a request to the exponent manager 110 to determine the floating-point value, a definition for a hierarchical shared-exponent floating-point data type, and an exponent value for the floating-point value.

[0031]

[0039] After receiving the floating-point values, the definition and the request for the hierarchical shared exponent floating-point data type, the exponent manager 110 determines the exponent value for each of the floating-point values, as shown in line 410. In this example, the exponent manager 110 determines the shared exponent value for each pair of original exponent values ​​as the shared exponent for level 0. In this example, the exponent manager 110 determines the shared exponent value for level 0 by determining the exponent value with the highest value for each pair of original exponent values ​​and using it as the shared exponent value for level 0. As shown in line 415, the highest exponent value among the exponent values ​​for the first pair of values ​​is -3, and the highest exponent value among the exponent values ​​for the second pair of values ​​is -2. The exponent manager 110 then determines the shared exponent for level 1 based on the shared exponent values ​​for level 0. In this example, the exponent manager 110 determines the shared exponent for level 1 by determining the shared exponent value for level 0 with the highest value and using it as the shared exponent value for level 1. Here, index manager 110 determines that the level 1 shared exponent value is −2, as shown in row 420. Next, index manager 110 determines a level 1 scale value for each pair of a level 0 shared exponent value and a corresponding level 1 shared exponent value. In this example, index manager 110 determines the level 1 scale value by determining the difference between the level 0 shared exponent value and the corresponding level 1 shared exponent value. If the difference is greater than 1, index manager 110 approximates the level 1 scale value to 1.

[0032]

[0040] For a first pair of level 0 shared exponent values ​​and corresponding level 1 shared exponent values, exponent manager 110 determines that the level 1 scale value is 1 (i.e., the difference between -2 and -3 is 1). For a second pair of level 0 shared exponent values ​​and corresponding level 1 shared exponent values, exponent manager 110 determines that the level 1 scale value is 0 (i.e., the difference between -2 and -2 is 0). The level 1 scale values ​​are shown in row 425. Row 430 depicts the total scale value, which is the same as the level 1 scale value. Row 435 depicts the total shared exponent scale value for each floating point value, which is the exponent value determined for the floating point value by subtracting the respective total scale value from the level 1 shared exponent value. Row 440 depicts the difference between the determined exponent value and the original exponent value for each floating point value. Row 445 depicts the quantized magnitude of the mantissa for the floating point value. In this example, a 3-bit mantissa is used, with the first bit used for the sign (not shown in table 400) and the second bit used for the magnitude. Row 450 shows the quantized exponent value for each of the floating-point values, which is the same as the shared-exponent scale value in row 435. Row 455 then depicts the decimal representation of the quantized floating-point values ​​after converting the original floating-point values ​​to the hierarchical shared-exponent floating-point data type.

[0033]

[0041] Upon determining the level 1 shared exponent and level 1 scale values, the exponent manager 110 sends them to the HASEFPDT manager 105. The HASEFPDT manager 105 sends the floating-point values ​​of line 405, the level 1 shared exponent, the level 1 scale value, a definition for the hierarchical shared-exponent floating-point data type, and a request to the mantissa manager 115 to determine the mantissa values ​​for the floating-point values. Once the mantissa manager 115 receives the data and the request, the mantissa manager 115 determines the mantissa values ​​for each floating-point value. If the exponent difference (shown in line 440) is greater than 1, the mantissa manager 115 quantizes the mantissa magnitude for each floating-point value to 0. Otherwise, the mantissa manager 115 uses the available bits (in this example, one bit for the sign and one bit for the magnitude) and each quantized exponent value (shown in line 450) to best approximate the original floating-point value. The mantissa manager 115 determines the quantized magnitude values ​​for each of the floating-point values ​​to be the values ​​shown on line 445. The mantissa manager 115 then sends the determined mantissa values ​​to the HASEFPDT manager 105.

[0034]

[0042] When the HASEFPDT manager 105 receives the mantissa value from the mantissa manager 115, the HASEFPDT manager 105 stores the mantissa value, the level 1 shared exponent, and the level 1 scale value in a data structure configured to store data for a hierarchical shared-exponent floating-point data type. FIG. 5 illustrates an example data structure 500 for storing the hierarchical shared-exponent floating-point data type shown in FIG. 4, according to some embodiments. The data structure 500 can be generated based on a definition for the hierarchical shared-exponent floating-point data type. In this example, the definition for the data type specifies four 2-bit mantissa values, two 2-bit level 1 scale values, and an 8-bit level 1 shared exponent value for four floating-point values. As depicted, the data structure 500 includes three attributes: a mantissa attribute, a level 1 scale attribute, and a shared exponent attribute. The mantissa attribute stores the 2-bit mantissa value determined for each of the floating-point values. The level 1 scale attribute stores the 2-bit level 1 scale value determined for each of the floating-point values. The shared-exponent attribute value stores the 8-bit level 1 shared-exponent value determined for the floating-point values. In this example, the value of the level 1 shared-exponent value is represented using a bias of 127. Thus, for this example, a value of 125 (i.e., 127+(-2)) is stored in the shared-exponent attribute.

[0035]

[0043] The examples described above with reference to Figures 1-5 illustrate how floating-point numbers can be converted into two different hierarchical shared-exponent floating-point data types. Those skilled in the art will appreciate that additional and / or different hierarchical shared-exponent floating-point data types can be used in some embodiments. For example, the hierarchical shared-exponent floating-point data types can represent a different number of floating-point values ​​that share an exponent (each of the above examples represents four floating-point values), use a different number of bits to represent the mantissa value, the level scale value, and the shared exponent value, use a different number of levels in the hierarchy of exponent values ​​(e.g., a different number of levels in the scale value), etc.

[0036]

[0044] In addition, the examples described above with reference to Figures 2-5 show how a hierarchy of shared exponent values ​​can be represented by a shared exponent value at the highest level and a level scale value. For example, the example shown in Figure 2 uses a hierarchy of shared exponents with three different levels: an exponent value at level 0, a shared exponent value at level 1, and a shared exponent value at level 2. As depicted by Figure 3, the hierarchy of shared exponent values ​​is represented by a shared exponent value at level 2, a scale value at level 2, and a scale value at level 1. The quantized exponent value of the floating-point value (the value of row 260) can be determined from the shared exponent value at level 2, a scale value at level 2, and a scale value at level 1. As another example, the example shown in Figure 4 uses a hierarchy of shared exponents with two different levels: an exponent value at level 0 and a shared exponent value at level 1. As shown by Figure 5, the hierarchy of shared exponent values ​​is represented by a shared exponent value at level 1 and a scale value at level 1. The quantized exponent value of the floating-point value (the value of row 450) can be determined from the shared exponent value at level 1 and a scale value at level 1. The HASEFPDT manager 105 determines the quantized exponent value in this manner when reading floating-point values ​​stored according to a hierarchical shared-exponent floating-point data type (e.g., determining quantized values ​​such as those shown in lines 265 and 455).

[0037]

[0045] Additionally, the examples described above with reference to Figures 2-5 determine the shared index value by selecting the highest value among the lower level indices / shared indexes. One skilled in the art will appreciate that additional and / or different statistical metrics can be used to determine a particular level's shared index value. Examples of such metrics include weighted averages, mean plus a defined number of standard deviations (e.g., 1 standard deviation, 2 standard deviations, 3 standard deviations, etc.).

[0038] Example Uses of Hierarchical Shared-Exponent Floating-Point Data Types

[0046] The examples described above describe how to create hierarchical shared-exponent floating-point data types. The following examples show one of many uses of these data types. FIG. 6 illustrates an example of multiplying hierarchical shared-exponent floating-point data type variables, according to some embodiments. Specifically, FIG. 6 illustrates a hardware architecture 600 for performing a dot product between two variables stored according to a hierarchical shared-exponent floating-point data type. In some embodiments, the hardware architecture 600 can be used to implement an artificial intelligence (AI) accelerator used for training neural networks and inference using neural networks. As shown, the hardware architecture 600 includes multipliers 605-1 to 605-n, shifters 610-1 to 610-n and 620-1 to 620-m, adders 615-1 to 615-m, 625-1 to 625-k and 630, and a truncate / round off module 635.

[0039]

[0047] In this example, the hardware architecture 600 is configured to receive two inputs A and B. Each of the inputs A and B is a hierarchical shared exponent floating-point data type variable (e.g., data structure 300, data structure 500, etc.) that stores n floating-point values. The mantissa value of the floating-point value of each variable is input to a corresponding multiplier 605. For example, the mantissa value of a first floating-point value of variable A and the mantissa value of a first floating-point value of variable B are input to multiplier 605-1, the mantissa value of a second floating-point value of variable A and the mantissa value of a second floating-point value of variable B are input to multiplier 605-2, the mantissa value of a third floating-point value of variable A and the mantissa value of a third floating-point value of variable B are input to multiplier 605-3, etc. As shown, the inputs to each of the multipliers 605 have a bit width of dw. Each multiplier 605 multiplies two input mantissa values ​​and outputs a product having a bit width of 2dw. The output of the multiplier 605 is input to a corresponding shifter 610, which is configured to perform a right shift based on a control input depicted at the top of the shifter 610. The control input to the shifter 610 is the sum of the level 1 scale values ​​of each of the inputs to the multiplier 605. The shifter 610 performs a number of right shift operations equal to the value of the control input. The output of the shifter 610 has a bit width of 2dw+3. The outputs of the two shifters 610 are input to a corresponding adder 615, which sums the two inputs to generate a sum having a bit width of 2dw+4. The process continues using the corresponding level scale values ​​to control the shifters (e.g., the sum of the level 2 scale values ​​is used to control the shifters 620-1 through 620-m). The number of levels is determined by log2(n). As mentioned above, n is the number of floating-point values ​​stored in the variable. The output of the last level has a bit width of 2dw+4log2(n).

[0040]

[0048] When the Truncate / Round module 635 receives the output from the last level, the Truncate / Round module 635 performs shift operations on the value of the output based on the sum of the shared exponent values ​​of each of the variables. For example, if the sum of the shared exponent values ​​is a positive value, the Truncate / Round module 635 performs a number of left shift operations on the value equal to the sum of the shared exponent values. If the sum of the shared exponent values ​​is a negative value, the Truncate / Round module 635 performs a number of right shift operations on the value equal to the absolute value of the sum of the shared exponent values. The Truncate / Round module 635 then truncates the value by a defined number of bits to generate the output value C.

[0041]

[0049] 7 illustrates a process 700 for creating hierarchical shared exponent floating-point data type variables according to some embodiments. In some embodiments, computing system 100 executes process 700. Process 700 begins at 710 by determining a first shared exponent value based on the exponent values ​​of each floating-point value of a first subset of the plurality of floating-point values. By way of example, and referring to FIGS. 1 and 4, exponent manager 110 may determine a level 0 shared exponent value based on the original exponent values ​​of a first pair of floating-point values.

[0042]

[0050] Next, process 700 determines a second shared exponent value based on the exponent values ​​of each floating-point value of the second subset of the plurality of floating-point values, at 720. By way of example, and with reference to Figures 1 and 4, exponent manager 110 may determine a shared exponent value for level 0 based on the original exponent values ​​of the second pair of floating-point values.

[0043]

[0051] The process 700 then determines a third shared exponent value based on the first and second shared exponent values ​​at 730. By way of example, and referring to Figures 1 and 4, the index manager 110 may determine a level 1 shared exponent value based on a level 0 shared exponent value.

[0044]

[0052] At 740, process 700 determines a first difference value based on the first shared exponent value and the third shared exponent value. By way of example, with reference to Figures 1 and 4, index manager 110 may determine a level 1 scale value for a first pair of level 0 shared exponent values ​​and corresponding level 1 shared exponent values. Process 700 then determines a second difference value at 750 based on the second shared exponent value and the third shared exponent value. By way of example, with reference to Figures 1 and 4, index manager 110 may determine a level 1 scale value for a second pair of level 0 shared exponent values ​​and corresponding level 1 shared exponent values.

[0045]

[0053] Next, the process 700 determines, at 760, a sign value and a mantissa value for each floating-point value of the plurality of floating-point values. By way of example, referring to FIGS. 1 and 4, the mantissa manager 115 may determine a quantized magnitude value for the mantissa as shown in row 445 of table 400. Finally, at 770, the process 700 stores the sign value and the mantissa value, the third shared exponent value, the first difference value, and the second difference value for each floating-point value of the plurality of floating-point values ​​in a data structure for a shared-exponent floating-point data type. By way of example, referring to FIGS. 1 and 5, the HASEFPDT manager 105 stores the mantissa value, the level 1 shared exponent, and the level 1 scale value in data structure 500.

[0046]

[0054] It is noted that the techniques described above can be implemented in a wide range of computer systems configured to process neural networks. FIG. 8 depicts a simplified block diagram of an exemplary computer system 800 (e.g., computing system 100) that can be used to implement the techniques described in the preceding disclosure. As shown in FIG. 8, computer system 800 includes one or more processors 802 that communicate with a number of peripheral devices via a bus subsystem 804. These peripheral devices may include a storage subsystem 806 (e.g., including a memory subsystem 808 and a file storage subsystem 810) and a network interface subsystem 816. Some computer systems may further include a user interface input device 812 and / or a user interface output device 814.

[0047]

[0055] Bus subsystem 804 may provide a mechanism for allowing the various components and subsystems of computer system 800 to communicate with each other as intended. Although bus subsystem 804 is shown generally as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses.

[0048]

[0056] Network interface subsystem 816 can serve as an interface for communicating data between computer system 800 and other computer systems or networks. Embodiments of network interface subsystem 816 may include, for example, Ethernet, Wi-Fi, and / or cellular adapters, modems (telephone, satellite, cable, ISDN, etc.), digital subscriber line (DSL) units, and / or the like.

[0049]

[0057] Storage subsystem 806 includes memory subsystem 808 and file / disk storage subsystem 810. Subsystems 808, 810 and other memories described herein are examples of non-transitory computer-readable storage media that may store executable program code and / or data that provides the functionality of embodiments of the present disclosure.

[0050]

[0058] The memory subsystem 808 contains a number of memories, including a main random access memory (RAM) 818 for storing instructions and data during program execution, and a read only memory (ROM) 820 in which fixed instructions are stored. The file storage subsystem 810 can provide persistent (e.g., non-volatile) storage for program and data files, and can include magnetic or solid state hard disk drives, optical drives with associated removable media (e.g., CD-ROM, DVD, Blu-Ray, etc.), removable flash memory based drives or cards, and / or other types of storage media known in the art.

[0051]

[0059] It should be understood that computer system 800 is exemplary and that many other configurations having more or fewer components than system 800 are possible.

[0052]

[0060] FIG. 9 illustrates a neural network processing system according to some embodiments. In various embodiments, a neural network according to the present disclosure can be implemented and trained in a hardware environment including one or more neural network processors. A neural network processor may refer to a variety of application specific integrated circuits (ASICs) or neural network processors, including, for example, various graphics processing units (GPUs) (e.g., GPUs for processing neural networks produced by Nvidia Corp®), field programmable gate arrays (FPGAs) (e.g., FPGAs for processing neural networks produced by Xilinx®), or hardware architectures optimized for neural network operations. In this exemplary environment, one or more servers 902, which may include the architecture shown in FIG. 8 above, may be coupled to multiple controllers 910(1)-910(M) over a communications network 901 (e.g., switches, routers, etc.). The controllers 910(1)-910(M) may also include the architecture shown in FIG. 8 above. Each controller 910(1)-910(M) may be coupled to one or more NN processors (e.g., processors 911(1)-911(N) and 912(1)-912(N)). The NN processors 911(1)-911(N) and 912(1)-912(N) may include various configurations of functional processing blocks and memory optimized for neural network processing (e.g., training or inference). The NN processors are optimized for neural network operations. The server 902 may configure the controller 910 to use NN models and input data to the models, which may be loaded and executed in parallel, for example, by the NN processors 911(1)-911(N) and 912(1)-912(N). The models may include, for example, layers and associated weights as described above. The NN processor may load the models and apply the inputs to generate output results. The NN processor may also implement, for example, training algorithms described herein.

[0053] Further Exemplary Embodiments

[0061] In various embodiments, the present disclosure includes systems, methods, and apparatus for providing a hierarchical shared exponent floating-point data type. The techniques described herein can be embodied in a non-transitory machine-readable medium storing a program executable by a computer system, the program including a set of instructions for performing the techniques described herein. In some embodiments, the system includes a set of processing units and a non-transitory machine-readable medium storing instructions that, when executed by at least one processing unit of the set of processing units, cause the at least one processing unit to perform the techniques described herein. In some embodiments, the non-transitory machine-readable medium can be, for example, a memory and can be coupled to, for example, one or more controllers or one or more artificial intelligence processors.

[0054]

[0062] The following techniques may be implemented alone or in different combinations, and may be further implemented with other techniques described herein.

[0055]

[0063] For example, in one embodiment, the present disclosure includes a non-transitory machine-readable medium storing a program executable by at least one processing unit of a device. The program includes a set of instructions for determining a first shared exponent value based on an exponent value of each floating-point value of a first subset of the plurality of floating-point values; determining a second shared exponent value based on an exponent value of each floating-point value of a second subset of the plurality of floating-point values; determining a third shared exponent value based on the first shared exponent value and the second shared exponent value; determining a first difference value based on the first shared exponent value and the third shared exponent value; determining a second difference value based on the second shared exponent value and the third shared exponent value; determining a sign value and a mantissa value for each floating-point value of the plurality of floating-point values; and storing the sign value and mantissa value, the third shared exponent value, the first difference value, and the second difference value for each floating-point value of the plurality of floating-point values ​​in a data structure for a shared-exponent floating-point data type.

[0056]

[0064] In one embodiment, the first shared index value, the second shared index value and the third shared index value form a hierarchy of shared index values.

[0057]

[0065] In one embodiment, the first shared exponent value and the second shared exponent value are exponent values ​​at a first level in a hierarchy of shared exponent values, and the third shared exponent value is an exponent value at a second level in the hierarchy of shared exponent values.

[0058]

[0066] In one embodiment, the hierarchy of shared index values ​​is represented by the third shared index value, the first difference value, and the second difference value.

[0059]

[0067] In one embodiment, the present disclosure further includes determining a first set of difference values ​​based on the first shared exponent value and the exponent value of each floating-point number of the first subset of the plurality of floating-point values, determining a second set of difference values ​​based on the second shared exponent value and the exponent value of each floating-point number of the second subset of the plurality of floating-point values, and storing the first set of difference values ​​and the second set of difference values ​​in a data structure for the shared exponent floating-point data type.

[0060]

[0068] In one embodiment, determining the first shared exponent value includes determining an exponent value of a floating-point value of a first subset of the plurality of floating-point values ​​that has a highest value, and using that exponent value as the first shared exponent value.

[0061]

[0069] In one embodiment, determining the second shared exponent value includes determining an exponent value of a floating-point value of a second subset of the plurality of floating-point values ​​that has a highest value, and using that exponent value as the second shared exponent value.

[0062]

[0070] In one embodiment, determining the third shared exponent value includes using the first shared exponent value as the third exponent value when determining that the first shared exponent value is greater than or equal to the second shared exponent value, and using the second shared exponent value as the third exponent value when determining that the first shared exponent value is less than the second shared exponent value.

[0063]

[0071] In one embodiment, the number of bits used to represent the mantissa value stored in the data structure for the shared exponent floating-point data type is less than the number of bits used to represent the third exponent value.

[0064]

[0072] The above description illustrates various embodiments of the present disclosure, along with examples showing how aspects of certain embodiments can be implemented. The above examples should not be considered the only embodiments, but are presented to illustrate the flexibility and advantages of certain embodiments, as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents can be employed without departing from the scope of the present disclosure, as defined by the claims.

Claims

1. A non-transitory machine-readable medium storing a program executable by at least one processing unit of a device, the program comprising: determining a first shared exponent value based on the exponent values ​​of each floating-point value of the first subset of the plurality of floating-point values; determining a second shared exponent value based on the exponent values ​​of each floating-point value of a second subset of the plurality of floating-point values; determining a third shared exponent value based on the first shared exponent value and the second shared exponent value; determining a first difference value based on the first shared exponent value and the third shared exponent value; determining a second difference value based on the second shared exponent value and the third shared exponent value; determining a sign value and a mantissa value for each floating point value of the plurality of floating point values; storing the sign value and the mantissa value, the third shared exponent value, the first difference value, and the second difference value for each floating-point value of the plurality of floating-point values ​​in a data structure for a shared-exponent floating-point data type; providing the data structure as an input to a set of hardware components configured to perform a set of operations on the plurality of floating-point values ​​based on the data structure for the shared-exponent floating-point data type and generate an output based on the set of operations, the set of hardware components including a set of multipliers for performing a multiplier operation on the mantissa values ​​of the plurality of floating-point values ​​to generate a plurality of products, a first set of shifters performing a shift operation on the plurality of products based on the first difference value and the second difference value to generate a first plurality of shifted values, a first set of adders adding the first plurality of shifted values ​​to generate a plurality of sums, a second set of shifters performing a shift operation on the plurality of sums to generate a second plurality of shifted values, and a second set of adders adding the second plurality of shifted values ​​to generate a set of sums; A non-transitory machine-readable medium that includes a set of instructions for performing the steps of the method.

2. 10. The non-transitory machine-readable medium of claim 1, wherein the first shared exponent value, the second shared exponent value, and the third shared exponent value form a hierarchy of shared exponent values.

3. 3. The non-transitory machine-readable medium of claim 2, wherein the first shared exponent value and the second shared exponent value are exponent values ​​at a first level in a hierarchy of the shared exponent values, and the third shared exponent value is an exponent value at a second level in the hierarchy of the shared exponent values.

4. 3. The non-transitory machine-readable medium of claim 2, wherein the hierarchy of shared exponent values ​​is represented by the third shared exponent value, the first difference value, and the second difference value.

5. The program is determining a first set of difference values ​​based on the first shared exponent value and the exponent value of each floating-point number of the first subset of the plurality of floating-point values; determining a second set of difference values ​​based on the second shared exponent value and the exponent value of each floating-point number of the second subset of the plurality of floating-point values; storing the first set of difference values ​​and the second set of difference values ​​in the data structure for the shared exponent floating-point data type; 10. The non-transitory machine-readable medium of claim 1, further comprising a set of instructions for:

6. 2. The non-transitory machine-readable medium of claim 1 , wherein determining the first shared exponent value comprises determining an exponent value of a floating-point value of the first subset of a plurality of floating-point values ​​that has a highest value, and using the exponent value as the first shared exponent value.

7. 2. The non-transitory machine-readable medium of claim 1 , wherein determining the second shared exponent value comprises determining an exponent value of a floating-point value of the second subset of a plurality of floating-point values ​​that has a highest value, and using the exponent value as the second shared exponent value.

8. 2. The non-transitory machine-readable medium of claim 1, wherein determining the third shared exponent value comprises: when determining that the first shared exponent value is greater than or equal to the second shared exponent value, using the first shared exponent value as the third exponent value; and when determining that the first shared exponent value is less than the second shared exponent value, using the second shared exponent value as the third exponent value.

9. 2. The non-transitory machine-readable medium of claim 1 , wherein a number of bits used to represent the mantissa value stored in the data structure for the shared-exponent floating-point data type is less than a number of bits used to represent the third exponent value.

10. determining a first shared exponent value based on the exponent values ​​of each floating-point value of the first subset of the plurality of floating-point values; determining a second shared exponent value based on the exponent values ​​of each floating-point value of a second subset of the plurality of floating-point values; determining a third shared exponent value based on the first shared exponent value and the second shared exponent value; determining a first difference value based on the first shared exponent value and the third shared exponent value; determining a second difference value based on the second shared exponent value and the third shared exponent value; determining a sign value and a mantissa value for each floating point value of the plurality of floating point values; storing the sign value and the mantissa value, the third shared exponent value, the first difference value, and the second difference value for each floating-point value of the plurality of floating-point values ​​in a data structure for a shared-exponent floating-point data type; providing the data structure as an input to a set of hardware components configured to perform a set of operations on the plurality of floating-point values ​​based on the data structure for the shared-exponent floating-point data type and generate an output based on the set of operations, the set of hardware components including a set of multipliers for performing a multiplier operation on the mantissa values ​​of the plurality of floating-point values ​​to generate a plurality of products, a first set of shifters performing a shift operation on the plurality of products based on the first difference value and the second difference value to generate a first plurality of shifted values, a first set of adders adding the first plurality of shifted values ​​to generate a plurality of sums, a second set of shifters performing a shift operation on the plurality of sums to generate a second plurality of shifted values, and a second set of adders adding the second plurality of shifted values ​​to generate a set of sums; The method includes:

11. The method of claim 10 , wherein the first shared exponent value, the second shared exponent value, and the third shared exponent value form a hierarchy of shared exponent values.

12. 12. The method of claim 11, wherein the first shared exponent value and the second shared exponent value are exponent values ​​at a first level in a hierarchy of the shared exponent values, and the third shared exponent value is an exponent value at a second level in the hierarchy of the shared exponent values.

13. The method of claim 11 , wherein the hierarchy of shared exponent values ​​is represented by the third shared exponent value, the first difference value, and the second difference value.

14. determining a first set of difference values ​​based on the first shared exponent value and the exponent value of each floating-point number of the first subset of the plurality of floating-point values; determining a second set of difference values ​​based on the second shared exponent value and the exponent value of each floating-point number of the second subset of the plurality of floating-point values; storing the first set of difference values ​​and the second set of difference values ​​in the data structure for the shared exponent floating-point data type; The method of claim 10 further comprising:

15. The method of claim 10, wherein determining the first shared exponent value includes determining an exponent value of a floating-point value of the first subset of a plurality of floating-point values ​​that has a highest value, and using the exponent value as the first shared exponent value.

16. The method of claim 10, wherein determining the second shared exponent value includes determining an exponent value of a floating-point value of the second subset of the plurality of floating-point values ​​that has a highest value, and using the exponent value as the second shared exponent value.

17. The method of claim 10, wherein determining the third shared exponent value includes using the first shared exponent value as the third exponent value when it is determined that the first shared exponent value is greater than or equal to the second shared exponent value, and using the second shared exponent value as the third exponent value when it is determined that the first shared exponent value is less than the second shared exponent value.

18. The method of claim 10, wherein the number of bits used to represent the mantissa value stored in the data structure for the shared exponent floating-point data type is less than the number of bits used to represent the third exponent value.

19. A set of processing units; a non-transitory machine-readable medium storing instructions; A system comprising: The instructions, when executed by at least one processing unit of the set of processing units, determining a first shared exponent value based on the exponent values ​​of each floating-point value of the first subset of the plurality of floating-point values; determining a second shared exponent value based on the exponent values ​​of each floating-point value of a second subset of the plurality of floating-point values; determining a third shared exponent value based on the first shared exponent value and the second shared exponent value; determining a first difference value based on the first shared exponent value and the third shared exponent value; determining a second difference value based on the second shared exponent value and the third shared exponent value; determining a sign value and a mantissa value for each floating point value of the plurality of floating point values; storing the sign value and the mantissa value, the third shared exponent value, the first difference value, and the second difference value for each floating-point value of the plurality of floating-point values ​​in a data structure for a shared-exponent floating-point data type; providing the data structure as an input to a set of hardware components configured to perform a set of operations on the plurality of floating-point values ​​based on the data structure for the shared-exponent floating-point data type and generate an output based on the set of operations, the set of hardware components including a set of multipliers for performing a multiplier operation on the mantissa values ​​of the plurality of floating-point values ​​to generate a plurality of products, a first set of shifters performing a shift operation on the plurality of products based on the first difference value and the second difference value to generate a first plurality of shifted values, a first set of adders adding the first plurality of shifted values ​​to generate a plurality of sums, a second set of shifters performing a shift operation on the plurality of sums to generate a second plurality of shifted values, and a second set of adders adding the second plurality of shifted values ​​to generate a set of sums; The system further comprises:

20. The instruction comprises: determining a first set of difference values ​​based on the first shared exponent value and the exponent value of each floating-point number of the first subset of the plurality of floating-point values; determining a second set of difference values ​​based on the second shared exponent value and the exponent value of each floating-point number of the second subset of the plurality of floating-point values; storing the first set of difference values ​​and the second set of difference values ​​in the data structure for the shared exponent floating-point data type; 20. The system of claim 19, further comprising:

Images