SYSTEM AND METHOD FOR INT9 QUANTIZATION

Abstract

To provide a method for converting data to be stored in a memory from a first format to a second format.SOLUTION: A method comprises the steps of: extending the number of bits of data to be stored in a double data rate (DDR) memory for one bit to form the extended data; determining whether the data to be stored in the DDR is signed data or unsigned data; and adding a sign value to the most significant bit of the extended data according to determination that the data is signed, and copying the data to a low-order bit of the extended data. The data is copied to the low-order bit of the extended data, and the most significant bit is set to an unsigned value, for example, zero according to determination that the data is unsigned. The extended data is stored in an on-chip memory (OCM) of a processing tile of a machine learning computer array.SELECTED DRAWING: Figure 3A

Description

[Background technology]

[0001] A hardware-based machine learning (ML) system typically includes multiple cores / subsystems (blocks and tiles), each with its own processing unit and on-chip memory (OCM). The ML system may process quantized numerical values ​​for various calculations. For example, quantized data stored in a memory unit, e.g., a double data rate (DDR) memory, may be transmitted to the processing tiles so that the data can be processed by the processing units for various ML operations.

[0002] Generally, floating point numbers (data) are converted to a quantized data format for storage, e.g., in DDR, and subsequent processing, e.g., by an ML system. Quantized formats may include, but are not limited to, signed integers, unsigned integers, etc. used in arithmetic logic unit (ALU) calculations. A mix of quantized format types is often used, e.g., by an ML system, for various calculations.

[0003] Unfortunately, currently, there is no mechanism to perform ALU calculations on a mixture of quantization format types without introducing an offset. Thus, the format type of each operand is tracked when a mixture of quantization format types is used in an ALU calculation, increasing the complexity and latency for the ALU calculation. Furthermore, in ML systems, integer values ​​may need to be rescaled before being input to a processing unit. However, mathematical division in a processor is often time-consuming and time-inefficient.

[0004] Related Art The foregoing examples and limitations associated therewith are intended to be illustrative and not exhaustive. Other limitations of the related art will become apparent upon reading of the specification and drawings. [Brief description of the drawings]

[0005] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is noted that, according to industry practice, various features have not been drawn to scale. In practice, dimensions of various features may be arbitrarily large or small for clarity of illustration.

[0006] [Figure 1] 1 illustrates an illustrative example of a hardware-based programmable architecture configured to support machine learning in accordance with an aspect of the present embodiment.

[0007] [Figure 2A] FIG. 2 illustrates quantization from one number format type to another implemented by the hardware-based programmable architecture in FIG. [Figure 2B] FIG. 2 illustrates quantization from one number format type to another implemented by the hardware-based programmable architecture in FIG. [Figure 2C] FIG. 2 illustrates quantization from one number format type to another implemented by the hardware-based programmable architecture in FIG. [Figure 2D] FIG. 2 illustrates quantization from one number format type to another implemented by the hardware-based programmable architecture in FIG.

[0008] [Figure 3A] 2 illustrates a flow diagram for converting stored data from a first format to a second format according to one aspect of the present embodiment. [Figure 3B] 2 illustrates a flow diagram for converting stored data from a first format to a second format according to one aspect of the present embodiment.

[0009] [Figure 4] 1 illustrates a diagram of an example architecture of a POD unit according to one embodiment of the present invention.

[0010] [Diagram 5] 1 illustrates a method for rescaling integer data in a machine learning operation according to one aspect of the presently described embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following description, many different embodiments or examples are provided to implement different features of the present subject matter. Specific examples of components and arrangements are described below to simplify the disclosure. These are, of course, merely examples and are not intended to be limiting. Furthermore, the present disclosure may repeat reference numerals and / or letters in the various examples. This repetition is for the purposes of brevity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations described.

[0012] Before the various embodiments are described in more detail, it should be understood that the embodiments are not limiting, since the elements in such embodiments may vary. It should also be understood that the specific embodiments described and / or illustrated herein have elements that can be easily separated from the specific embodiment and, optionally, can be combined with any of several other embodiments or can be replaced with elements in any of several other embodiments described herein. It should also be understood that the terminology used herein is for the purpose of describing certain concepts, and the terminology is not intended to be limiting. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood in the art to which the embodiments belong.

[0013] A new programmable hardware architecture for machine learning (ML) is proposed, including at least a host, a memory, a core, a data streaming engine, an instruction streaming engine, and an interface engine. The memory is configured to store floating-point numbers in a quantized format, including but not limited to int8, uint8, etc. According to some embodiments, the quantized data stored in the memory is converted to an int9 format to represent different quantized data format types, e.g., uniformly int8, uint8, etc., providing symmetric quantization of the data (i.e., the quantization is symmetric about zero) while also eliminating the need to perform offset calculations. It is appreciated that converting the data to an int9 format type simplifies the complexity by allowing the inference engine to perform ALU calculations on operands of homogeneous int9 format type without the need to keep track of the format type for the quantized operands, which also results in faster processing times.

[0014] In some embodiments, when data is read from a memory unit, e.g., a DDR, an 8-bit number is converted to an int9 format type based on, as a non-limiting example, whether the number stored in the memory unit is an int8 or a uint8. In some embodiments, 9 bits are used, the int8 format type is a sign-extended version of the int9 format type, and the uint8 format type is copied into the least significant bits of the 9-bit data, whose most significant bit (i.e., bit order 9) is unsigned, e.g., set to zero. It is understood that 8 bits are not enough to store the full int9 range, so a software component may ensure that an int9 value written to a memory unit, e.g., an 8-bit DDR, is within the proper range of an int8, uint8, etc.

[0015] In some embodiments, the software component is configured to perform an operation that restricts the range of the int9 number to be within the range of an int8, uint8, etc., or alternatively, perform an operation on the int9 number to represent it as two int9 numbers, one in the int8 range and one in the uint8 range. Representing the int9 number as two int9 numbers, one in the int8 range and one in the uint8 range, allows the least significant bits of the int9 number to be copied into the 8-bit DDR while preserving information.

[0016] In some embodiments, the cores of the programmable hardware architecture are configured to interpret ML commands / instructions on ML operations and / or data received from the host and adjust the streaming and inference engine activities based on the data in the received ML commands. The inference engines may include a dense arithmetic engine and an irregular arithmetic engine. The dense arithmetic engine is an engine optimized for efficiently processing dense data with regular operations, e.g., matrix operations, e.g., multiplication, matrix manipulation, tanh, sigmoid, etc. On the other hand, the irregular arithmetic engine is an engine optimized for efficiently processing sporadic data with irregular operations, e.g., memory transpose, operations on irregular data structures (e.g., trees, graphs, and priority queues). According to some embodiments, the cores may adjust some of the instructions received from the host to be processed. In some embodiments, the cores may be general-purpose processors, e.g., CPUs.

[0017] In some embodiments, the core is specifically configured to split multiple ML commands between the core and the inference engine for efficient execution. The ML commands and associated data executed by the inference engine are transmitted from the core and memory to the instruction streaming engine and data streaming engine for efficient streaming to the inference engine. As described above, data read from the memory unit is converted to int9 format. The data and instruction streaming engine is configured to send one or more data streams and ML commands to the inference engine in response to the programming instructions received from the core. The inference engine is configured to process the instruction / data streams received from the data / instruction stream engine for ML operations in accordance with the programming instructions received from the instruction / data streaming engine.

[0018] It is understood that data input to the dense arithmetic engine of the inference engine may need to be rescaled before certain operations, e.g., tanh, sigmoid, etc. To rescale in an efficient manner, data input in int32 format is multiplied by an integer scale value and shifted afterwards. To achieve the highest possible accuracy and lowest error in the calculations, the relationship between the integer scale value and the shift value is obtained based on the size of the register that stores the integer data, e.g., int32.

[0019] Now referring to FIG. 1, an example diagram of a hardware-based programmable system / architecture 101 configured to support machine learning is shown. Although the diagram shows the components as functionally separated, such a representation is for illustrative purposes only. It is clear that the components depicted in this diagram can be arbitrarily combined and divided into separate software, firmware and / or hardware components. It is further clear that such components, regardless of how they are combined or divided, can run on the same host or multiple hosts, and the multiple hosts can be connected by one or more networks. Each of the engines in the architecture 101 is a dedicated hardware block / component that includes one or more microprocessors and on-chip memory units that store data and software instructions programmed by a user for various machine learning operations. As will be described in detail below, when the software instructions are executed by the microprocessor, each of the hardware components becomes a dedicated hardware component for training a certain machine learning function. In some embodiments, the architecture 101 is a single chip, e.g., a system-on-chip (SOC).

[0020] In the example of FIG. 1, the architecture 101 may include a host 110 coupled to a memory (e.g., DDR) 120 and a core engine 130. The memory 120 may be coupled to a direct memory access (DMA) engine (not shown) and a network interface controller (NIC) (not shown) for receiving external data. The memory 120 may be internally connected to a data streaming engine 140. The core 130 is coupled to an instruction streaming engine 150, which is coupled to the data streaming engine 140. The core 130 is also coupled to a general purpose processor 165. In some embodiments, the general purpose processor 165 may be part of the core 130. The instruction streaming engine 150 and the data streaming engine 140 are coupled to an inference engine 160, which includes a dense arithmetic engine 161 and an irregular arithmetic engine 163. It is understood that the inference engine 160 may include arrays for performing various calculations. It is understood that any description of arrays for performing various calculations in inference engine 160 is for illustrative purposes and should not be construed as limiting the scope of the embodiments. For example, in some embodiments, arrays for performing various calculations may exist outside of inference engine 160.

[0021] It is understood that the external data may be in a floating point format, for example, 32-bit floating point. Thus, when the data is stored in the memory 120, for example, in an 8-bit DDR, the data may be converted to an integer format type, for example, int8, uint8, etc. It is understood that uint8 ranges from 0 to 255, while int8 ranges from -128 to 127. Meanwhile, int9 ranges from -256 to 255, and thus both int8 and uint8 can be represented without any offset calculation. Using int9 as the uint8 range and int8 range allows the data to be copied to a standard 8-bit DDR. It is understood that the description of the use of 32-bit floating point and 8-bit DDR is for illustrative purposes and should not be construed as limiting the scope of the embodiments. The floating point data is finally quantized to int9 instead of int8 or uint8. Furthermore, the int9 range covers both positive and negative values, resulting in a zero offset, further simplifying rescaling of int9 numbers in the ML system. Thus, when data is being read from memory 120, e.g., 8-bit DDR, the data is converted to int9 format. Once the data is converted to int9 format, it is understood that there is no need to track the types of operands when a mix of different format types is used in a calculation. For example, using int9 eliminates the need to track whether the operands in the calculation being performed are int8, uint8, etc.

[0022] It is further understood that in some embodiments, the memory 120, e.g., DDR, may store floating point numbers, e.g., 32-bit floating point, as four 8-bit values. Thus, when data is read from the memory 120, e.g., 8-bit DDR, into the on-chip memory, a quantization is performed in either the general-purpose processor 165 or the irregular arithmetic engine 163 from the 32-bit floating point to an int9. In some embodiments, registers in the general-purpose processor 165 and / or the irregular arithmetic engine 163 store 32-bit widths that hold the 32-bit floating point values. Thus, for use in the ML system, the floating point numbers may be converted to int9 numbers. However, the 32-bit floating point numbers are first scaled to convert to an int9 format. For example, a suitable scale may be as follows: scale = (upper float range - lower float range) / (upper int9 range - lower int9 range) = (end - (-end)) / (255 - (-255)) = 2end / (2(255)) = end / 255.

[0023] It is understood that the same scale may be used if the lower range of int9 is extended to include -256. It is understood that FIG. 2A shows symmetric quantization and mapping of 32-bit floating point to the full range of int9, while FIG. 2B shows the mapping of FIG. 2A to include -256 of int9. As shown, the same scale may be used for both FIG. 2A and FIG. 2B. FIG. 2C shows mapping of 32-bit floating point to int9 and a representation of an int8 within the int9 range. It is understood that the scale for the representation of an int8 within the int9 range is different than the scale determined above. In some embodiments, the scale for representing an int8 using 9 bits may be as follows: scale=(upper float range-lower float range) / (upper int8 range-lower int8 range)=(end-(-end)) / (127-(-127))=end / 127.

[0024] Figure 2D shows the mapping of 32-bit floating point to int9 and a representation of uint8 in the int9 range, it being understood that the representation of uint8 in the int9 range has the same scale as that of Figures 2A and 2B.

[0025] In some embodiments, when transferring data from memory 120 to an array, such as inference engine 160, the transferred data is sign extended or zero extended depending on whether the transferred data is int8 or uint8. That is, the data is converted from one format type, such as int8, uint8, etc., to another format type, such as int9. As a non-limiting example, when converting data from int8 or uint8 to int9 format, 8-bit data is converted to 9-bit data by extending the number of bits by 1 bit. It is determined whether the data being converted is signed, such as int8, or unsigned, such as uint8. If the data being converted is signed, the most significant bit of the 9-bit data for int9 is sign extended, and if the data being converted is unsigned, the most significant bit of the 9-bit data for int9 is set to zero. It is understood that the int8 or uint8 is copied directly into the lower bits (i.e., the lower 8-bit order) of the int9 data. It is understood that the int9 data may be referred to as extended data compared to the int8 or uint8 format type. The extended data, which is int9 format type in this example, is stored in the inference engine 160 to be operated on. In some embodiments, the extended data may be stored in an on-chip memory (OCM) of the inference engine 160 to be processed by the processing tiles of the ML computer array. It is understood that in some embodiments, a floating point number, e.g., a 32-bit floating point, may be converted to an integer representation, e.g., int9. In one exemplary embodiment, the floating point number is appropriately quantized and scaled to be converted to the int9 format type, as illustrated in FIGS. 2A-2D. As illustrated, one scaling value may be used to represent a floating point number relative to an int8 in the int9 format type, while a different scaling value may be used to represent a floating point number relative to a uint8 in the int9 format type.It is understood that a 16-bit floating point number stored in memory unit 120, e.g., DDR, remains the same as when it is stored from memory unit 120 to the OCM of inference engine 160. As a non-limiting example, the lowest 7 bits of a 16-bit floating point number are the same as the lowest 7 bits of its mantissa, an 8th bit is extended but not used in any operation, the remaining mantissa bits are the 9th and 10th bits followed by the exponent, and the sign bit is followed by an additional extension bit that is not used in any operation.

[0026] It is understood that the inference engine 160 may include a number of processing tiles arranged in a two-dimensional array of a number of rows and columns, e.g., 8 rows by 8 columns. Each processing tile may include at least one OCM, one POD unit, and one processing engine / element (PE), where the OCM in the processing tile is configured to receive data from the data streaming engine 140 in a streaming manner. It is understood that the received data may be in int9 format, as explained above. The OCM allows efficient local access to the data per processing tile. The processing units, e.g., PODs and PEs, are configured to perform highly specialized tasks, e.g., dense or sparse computation of ML operations, respectively, on the data received at the OCM.

[0027] It is understood that the OCM of each processing tile may receive data in int9 format type for various ALU operations associated with the ML operations. It is understood that in some embodiments, the format type of the data stored in memory 120, e.g., whether signed or unsigned, is tracked such that the appropriate instructions may be scheduled to be streamed for execution by the appropriate processing unit, e.g., the respective POD / PE of the processing tile. That is, the various ALU operations are performed by the processing tiles on the data received in int9 format. The data received in int9 format may be operands of the various ALU operations. The results of the various ALU operations in the int9 format type may be stored in their respective OCMs.

[0028] In some embodiments, the inference engine 160 includes a dense arithmetic engine 161 that is optimized for efficient processing of dense data, e.g., data received from memory 120 in int9 format, with regular operations, e.g., matrix operations, e.g., multiplication, matrix manipulation, tanh, sigmoid, etc. However, the inference engine 160 may also include an irregular arithmetic engine 163 that is optimized for efficient processing of sporadic data, e.g., int9 format type, with irregular operations, e.g., memory transpose, addition operations, operations on irregular data structures (e.g., trees, graphs, and priority queues). According to some embodiments, the core 130 may condition some of the instructions received from the host 110 to be processed by a general-purpose processor 165, e.g., a CPU, etc.

[0029] In some embodiments, core 130 is configured to execute any software code written through a common high-level language. Core 130 is configured to process a number of performance-non-critical operations, such as data / instruction preparatory work, data gathering, data mapping, etc. In some embodiments, core 130 may be configured to classify received ML commands into performance-critical and non-critical operations / tasks, such that performance-non-critical operations can be processed by core 130 and performance-critical operations (e.g., matrix multiplication) can be processed by inference engine 160. That is, core 130 is configured to divide the ML commands between core 130 and inference engine 160 for their efficient execution. In some embodiments, core 130 may be configured to assign / divide the ML commands (also referred to as tasks or subtasks) to various components, such as inference engine 160, for processing. In some embodiments, core 130 is configured to allocate one or more locations in memory 120 to store tasks / commands, data, results after data has been processed, etc., for access and use by core 130 or other components in architecture 101, such as inference engine 160. Thus, core 130 and inference engine 160 are configured to execute entire ML algorithms and operations thereon, instead of relying on or requiring host 110 to execute certain ML commands or operations. By supporting and executing entire ML operations in programmable hardware architecture 101, core 130 eliminates the performance overhead of transferring data to host 110 and back to execute any unsupported ML operations, reducing the burden on host 110 and achieving higher performance.

[0030] In some embodiments, ML commands and associated data, e.g., in their int8 format, to be executed by inference engine 160 are transmitted from cores 130 and memory 120 to instruction streaming engine 150 and data streaming engine 140 for efficient streaming to inference engine 160. In some embodiments, data / instruction streaming engines 140-150 are configured to send one or more data streams and programming instructions to inference engine 160 in response to ML commands received from cores 130. It will be appreciated that in some embodiments, the format type of data stored in memory 120, e.g., whether signed or unsigned, is tracked such that appropriate instructions can be scheduled to be streamed to dense arithmetic engine 161 and / or irregular arithmetic engine 163 of inference engine 160. That is, various ALU operations are performed by engines within inference engine 160 on data received in int9 format. Data received in int9 format may be operands of various ALU operations. Results of various ALU operations in the int9 format type may be stored in dense arithmetic engine 161 and / or irregular arithmetic engine 163 of inference engine 160. In some embodiments, the results may be stored in an appropriate OCM of a processing tile of the ML computer array.

[0031] It is understood that in some embodiments, the results of the ALU operations in the inference engine 160 are stored in a memory component, e.g., the OCM, of each processing tile in the inference engine 160. The results stored in the inference engine 160 may be transmitted for storage to the memory unit 120, e.g., a DDR. However, before storing the result, if the value of the result exceeds an upper limit, e.g., a maximum, of the format type of the data in the memory unit 120, the value may be adjusted to an upper range for the data, and if the value of the result is less than a lower range of the memory unit 120, the value may be adjusted to a lower range for the data, e.g., a minimum. It is understood that when storing the result from the OCM of each processing tile to the memory unit 120, the most significant bits of the int9 result are dropped.

[0032] It is understood that in some embodiments, the results of the processing stored in each OCM may be transmitted for storage back to memory unit 120, e.g., DDR. However, before storing the results, if the value of the result exceeds an upper limit, e.g., a maximum, for the format type of the data in memory unit 120, the value may be adjusted to the upper range for the data, and if the value of the result is less than the lower range of memory unit 120, the value may be adjusted to the lower range for the data, e.g., a minimum. That is, the data may be clamped to be within a proper range, e.g., an int8 range, a uint8 range, etc. It is understood that when storing the results from the OCM of each processing tile to memory unit 120, the most significant bits of the int9 result are dropped. It will further be appreciated that when transferring data from an inference engine, e.g., the respective OCM of inference engine 160, to memory unit 120, e.g., DDR, the software module may track whether the data stored in memory unit 120 was signed or unsigned, so that an int9 data format type can be correctly interpreted as the appropriate format type, e.g., int8 for data that was in int8 format in memory unit 120, uint8 for data that was in uint8 format in memory unit 120, etc.

[0033] 3A and 3B, an example process is shown that supports converting data stored in a memory, e.g., a DDR, from a first format, e.g., int8, uint8, floating point, etc., to a second format type, e.g., int9, as described in FIGS. 1-2D. Although the diagram shows functional stages in a particular order for purposes of illustration, the process is not limited to any particular order or arrangement of stages. Those skilled in the art will appreciate that the various stages depicted in the diagram may be omitted, rearranged, combined, and adapted in various ways.

[0034] As shown in FIG. 3A and FIG. 3B, in step 310, the number of bits stored in the memory unit 120, e.g., DDR, is extended by one bit to form extended data, e.g., int9. Thus, an int8 or uint8 containing 8 bits is extended to 9 bits. It is understood that in some embodiments, the data stored in the memory unit 120 is a floating point number. In step 320, it is determined whether the data stored in the memory 120 is signed, e.g., int8, or unsigned, e.g., uint8. In response to determining that the data is signed in step 330, the extended data is sign extended. Meanwhile, in response to determining that the data is unsigned in step 340, the most significant bit of the extended data is set to zero. In step 350, the data is copied to the lower bits, and thus to all order bits except the most significant bit. In step 360, the extended data is copied to the inference engine 160, e.g., the OCM of the inference engine 160. It is understood that in step 370, it is tracked whether the data stored in the memory unit 120, e.g., DDR, is signed or unsigned, and accordingly in step 380, the appropriate instruction for the expanded data is scheduled. In step 382, ​​various ALU operations may be performed on the expanded data. In step 384, the result of the ALU operation is stored in the OCM. In step 386, the result of the ALU operation stored in the OCM is also stored / copied to the memory unit 120, e.g., DDR. In step 388, the most significant bits of the result are dropped before storing the result from the OCM to the DDR. It is understood that, optionally, in step 390, the value of the ALU result may be adjusted before storing from the OCM to the memory unit 120, e.g., DDR, based on the range of the number format type stored in the memory unit 120. For example, if the resulting value stored in the OCM exceeds the upper range of the numeric type stored in the memory unit 120, e.g., int8 or uint8, the result is adjusted and changed to the maximum or upper range of the numeric value and therefore the int8, uint8, etc.

[0035] FIG. 4 shows a diagram of an example architecture of a POD. It is understood that the number of components, the size of components and the number of bits, matrix sizes, etc. shown in FIG. 4 are for illustrative purposes and are not intended to limit the scope of the embodiments. In the following description, matrix multiplication is used as a non-limiting example, but it is understood that the POD is also configured to perform other types of dense computation tasks of ML operations. In the example of FIG. 4, the POD includes a computation POD instruction control 699 configured to control the loading of data / instructions to various components, e.g., registers, tanh / sigmoid unit 614, etc. It is understood that the POD includes a matrix multiplication block 602, which is a two-dimensional array with X rows and Y columns, and each element / cell in the array has a certain number of registers (e.g., MIPS or microprocessors without interlocked pipeline stages). The matrix multiplication block 602 is configured to multiply two matrices, matrix A consisting of X rows and Z columns and matrix B consisting of Z rows and Y columns, to generate matrix C consisting of X rows and Y columns. It is understood that even if the data stored in the memory unit 120 is in a different type of format, for example, int8, uint8, floating point, etc., the data to be multiplied may be int9 format type stored in each OCM.

[0036] In the example of FIG. 4, the POD further includes three types of registers, namely, A registers 604, B registers 606, and C registers 608, which provide matrix data to the matrix multiplication block 602 for matrix multiplication. The A registers 604 include a bank of registers, e.g., m registers, each configured to maintain one row / column of the A matrix that is provided to a column of the array of the matrix multiplication block 602. Each A register may have multiple entries, e.g., k elements, each having a certain number of bit widths and supporting one read or write operation per cycle. It is understood that even if the data stored in the memory unit 120 is of a different format type, e.g., int8, uint8, float type, etc., the data may be in each register in an int9 format type. That is, the data is converted from one format of the memory unit 120 to a different format type, e.g., int9, to be stored in the OCM of each of the PEs and / or processing tiles used in the ALU computation of the POD operation. The entries allow each A register to pre-fetch the next portion of the A matrix before they are needed for calculation by the matrix multiplication block 602. The B register 606 includes a bank of registers, e.g., n registers, each configured to hold one row / column of the B matrix that feeds into a row of the array of the multiplication block 602. Like the A register 604, each B register may have multiple entries, e.g., k elements, each having a certain number of bits width and supporting one read or write operation per cycle. The entries allow each B register to pre-fetch the next portion of the B matrix before they are needed for calculation by the matrix multiplication block 602. The C register 608 is configured to hold the matrix--the result of the multiplication--the C matrix--generated by the multiplication block 602. The C register 608 includes multiple banks, each configured to hold one row / column of the C matrix. The C matrix is ​​configured to have m×n elements.

[0037] During the matrix multiplication process, the matrix multiplication block 602 is configured to read the elements of matrices A and B from the OCM only once into the A and B registers, respectively (instead of reading each row or column of the matrix), thus saving memory access time to the OCM. Specifically, each matrix multiplication operation has a unique structure, where a row of a first matrix is ​​multiplied with every column in a second matrix, and a column in a second matrix is ​​multiplied with every row in a first matrix. As the matrix multiplication block 602 performs the matrix multiplication operation, each row of the A register 604 remains the same, while the columns of the B register 606 are provided to the matrix multiplication block 602 one at a time to be multiplied with the rows in the A register 604. At the same time, each column of the B register 606 remains the same, while the rows of the A register 604 are provided to the matrix multiplication block 602 one at a time to be multiplied with the columns of the B register 606. Thus, matrix multiplication block 602 is configured to simultaneously multiply each row of the first matrix with every column of the second matrix, and each column of the second matrix with every row of the first matrix, with the outputs from these multiplications being accumulated and stored in the C register until the matrix multiplication process is completed.

[0038] As shown in the example of FIG. 4, A register 604, B register 606 and C register 608 are each associated with a corresponding OCM streamer 603, 605 or 607, each of which is programmed and configured to stream data from the OCM to the corresponding register to ensure that the matrix multiplication operation can be performed by the matrix multiplication block 602 in a simplified manner. Each OCM streamer has an address range of the OCM to read from and a stride to track for the next read. The A or B type register is configured to send a next line ready signal per bank to its corresponding streamer, where the bit pattern of the signal indicates which bank requires the next line of data. The corresponding streamer of the A or B register responds to the read signal by sending the corresponding line of data from the OCM to the register. The streamer sends a done signal to its corresponding register when it has sent the last line of data to be transmitted. When all of the banks of registers have a line of data, the A or B register sends a ready signal to the matrix multiplication block 602 that the next set of A or B registers is ready to be loaded into the matrix multiplication block 602 for matrix multiplication. In some embodiments, each register bank has a valid bit that tells the matrix multiplication block 602 which values ​​are valid and should be operated on.

[0039] When the matrix multiplication is complete, e.g., the end of a row of the A matrix and the end of a column of the B matrix are reached, the matrix multiplication block 602 notifies the C registers 608 that all accumulations in the C registers 608 entries are complete and the entries are ready to be written back to the OCM via their corresponding streamers 607. Each bank of C registers 608 then transmits data to the OCM. If the OCM is not ready to accept data from a bank of C registers 608, the transmission is delayed and retried in the next cycle until the PE is ready to accept data from the bank. In some embodiments, the C registers 608 are preloaded with data or reset to zero before the next set of accumulations during the next matrix multiplication operation. Such preloading allows for biasing to be added as part of the next matrix multiplication. In some embodiments, each PE is configured to accept the output C matrix from the matrix multiplication block 602 of the POD, process it, and write it to the OCM.

[0040] According to one example, the results of the processing stored in each OCM may be transmitted for storage back to memory unit 120, e.g., DDR. However, before storing the results, if the value of the result exceeds an upper limit, e.g., a maximum, for the format type of the data in memory unit 120, the value may be adjusted to an upper range for the data, and if the value of the result is less than a lower range of memory unit 120, the value may be adjusted to a lower range, e.g., a minimum, for the data. That is, the data may be clamped to be within a proper range, e.g., an int8 range, a uint8 range, etc. It is understood that when storing the results from the OCM of each processing tile to memory unit 120, the most significant bits of the int9 result may be dropped. It will further be appreciated that when transferring data from an inference engine, e.g., the OCM of each of inference engines 160, to a memory unit 120, e.g., a DDR, the software module may track whether the data stored in memory unit 120 was signed or unsigned so that an int9 data format type can be correctly interpreted as the appropriate format type, e.g., int8 for data that was in int8 format in memory unit 120, uint8 for data that was in uint8 format in memory unit 120, etc.

[0041] In some embodiments, the inference engine 160 is configured to fuse / merge these later matrix multiplication operations by each PE with the matrix multiplication operations by the corresponding POD, so that these later matrix multiplication operations are performed immediately on the output from the matrix multiplication block 602, without first transmitting and storing the output to the OCM and then reading the C matrix from the OCM again for these later matrix multiplication operations. By bypassing the round trip to the OCM, the fusion of the later matrix multiplication operations with the matrix multiplication operations saves time and improves the efficiency of the inference engine 160. For example, it is understood that in some embodiments, additional regular operations, such as rectified linear unit (ReLU), quantization, etc., may be required on the output C matrix. Thus, a switching mechanism may be integrated within the POD architecture to determine whether additional regular operations are required, and if so, the output C matrix may be manipulated instead of writing it to another memory location. For example, if a rectified linear operation is required, the output C matrix is ​​streamed to a ReLU unit 601, which is configured to perform a ReLU operation on the C matrix. Similarly, if quantization is required, the output C matrix or the output of the ReLU unit 601 is streamed to a quantization unit 612 that is configured to quantize the C matrix or the results from the ReLU operation.

[0042] In some embodiments, the scale, shift and / or offset values ​​required for the quantization / requantization operations may be statically set by the core 130 and may be different for different ML operations. In some embodiments, these values ​​may be part of the ML model downloaded to the core, and values ​​corresponding to the ML operations may be read from the model and written to the appropriate registers before the quantization operation begins. It is understood that the requantization performs a rescaling of the output values ​​stored in the C register 608 for input to the quantization 612 and / or tanh / sigmoid unit 614 and later for direct storage in their respective OCM blocks. It is understood that the requantization may be performed on the output data, e.g., the C register 608 in this example, but in other examples, the requantization may be performed on other outputs from other registers. Thus, performing the requantization on the data stored in the C register 608 is for illustrative purposes and should not be construed as limiting the scope of the embodiments. In some embodiments, a single scaling value is applied to all elements of the output. It is understood that the scaling operation, which is a division operation, may be replaced with an integer multiplication and shift operation. It is further understood that the relationship between the value of the integer multiplication (also referred to as the integer scale value) and the shift value determines the precision and error in the system. In some embodiments, the relationship between the integer scale value and the shift value is obtained, and the largest possible value for the integer scale value and its corresponding scale value is selected based on the size of the register that stores the result of the multiplication (the multiplication of the output from the C register 608 with the integer scale value). In some embodiments, the output from the C register 608 may be denoted as V, and the quantization multiplier may be denoted as x, where x can be greater than or less than 1. It is understood that the relationship between the integer scale value and the shift value determines the quantization multiplier. The relationship between the integer scale value and the shift value is approximately given by the following mathematical formula (1): x~integer scale value / (2 シフト値 ) (1). Therefore, Integer scale value = int(x*2シフト値 ) (2). It is understood that the largest integer scale value is limited by the size of the register that holds the result of the integer multiplication, and therefore the integer scale value limits the output of the C register 608, e.g., the V value. For example, if V is 32 bits and the register size is 64 bits, then the integer scale value must be smaller than the largest 32-bit integer, or it will overflow. That is, the largest possible value is 2147483647. It is understood that the largest possible value for other sizes may be different, and the examples provided above are for illustrative purposes only and are not intended to limit the scope of the embodiments. Therefore, the condition shown in the following formula (3) will be met: integer scale value / largest possible value<1 (3)

[0043] In some embodiments, equations (2) and (3) are repeated throughout to obtain the largest possible integer scale value. Initially, the shift value is 0, and with each iteration, the shift value is incremented by a value, e.g., 1, 2, 5, 6, 7, 11, etc. The shift value determines the possible integer scale values, and another iteration is performed as long as the condition specified by equation (3) holds. The process is repeated until equation (3) is no longer true, at which point the previous shift value and its corresponding integer scale value are selected. It is understood that any of the previous shift values ​​and their corresponding integer scale values ​​may be selected, even though the largest previous integer scale value and its corresponding scale value provides the highest precision given the size of the register. The above process of selecting the largest possible integer scale value and its corresponding scale value is shown in Python. [Table 1]

[0044] It is understood that once the integer scale value and its corresponding scale value are selected, a quantization / requantization operation may be performed. The output of the C register 608 is multiplied by the integer scale value. The result of the multiplication is shifted by the shift value as selected above to form the scaled integer data. Once the data is scaled, additional operations may be performed, such as a tangent operation, a sigmoid operation, a rounding operation, a clipping / clamping operation, etc. In some embodiments, a rounding operation is performed by taking into account the most significant bit that is dropped due to the shift operation, and rounding the remaining result based on the most significant bit that was dropped. It is understood that the scaled integer data may be further adjusted based on the range for the integer data. For example, if the integer data stored in the memory unit 120 is an int8, and if the scaled integer data exceeds the upper limit of the int8, the scaled integer data is changed and adjusted to the maximum or upper limit of the int8. Similarly, if the integer data stored in the memory unit 120 is a uint8, and if the scaled integer data exceeds the upper limit of the uint8, the scaled integer data is changed and adjusted to the maximum or upper limit of the uint8. On the other hand, if the scaled integer data has a value lower than the minimum or lower range of data stored in memory unit 120, e.g., an int8 or uint8, the scaled integer data is adjusted and changed to the minimum or lower range of integer data in memory unit 120.

[0045] Now, referring to FIG. 5, a method for rescaling integer data in a machine learning operation is shown. It is understood that the method illustrated in FIG. 5 is a method flow for the operation as described in FIG. 4. In step 510, a relationship between the integer scale value and the shift value is determined as described in equation (1). In step 520, the shift value is iteratively increased and its corresponding integer scale value is obtained with respect to equation (2) until the condition shown in equation (3) is no longer true and thus the value is greater than or equal to 1. In step 530, a shift value and its corresponding integer scale value are selected before equation (3) is no longer true. It is understood that in some non-limiting examples, steps 510 to 530 are performed during the compilation stage and before any inference by the inference engine 160. In step 540, an integer value, for example in int32 format, is received from, for example, the C register 608. In step 550, the received integer value is multiplied by the selected integer scale value. In step 560, the result of the multiplication is shifted by a shift value corresponding to the selected integer scale value. In step 570, further operations may be performed, such as tanh, sigmoid, rounding, clipping, clamping, etc. In step 580, the value of the scaled integer data may be adjusted based on the range of the integer data stored in the memory unit 120. For example, if the int8 type data is stored in the memory unit 120, e.g., in a DDR, then if the scaled integer data exceeds the upper limit of the int8 data type, the scaled integer data is changed to the maximum or upper limit value of the int8 type data. Similarly, if the uint8 type data is stored in the memory unit 120, e.g., in a DDR, then if the scaled integer data exceeds the upper limit of the uint8 data type, the scaled integer data is changed to the maximum or upper limit value of the uint8 type data. On the other hand, if the scaled integer data is less than the lower limit of the int8 data type, the scaled integer data is changed to the minimum or lower limit of the int8 data type stored in the memory unit 120, e.g., in a DDR.Similarly, if the scaled integer data is smaller than the lower limit of the uint8 data type, the scaled integer data is changed to the minimum or lower limit of the uint8 data type stored in the memory unit 120, e.g., DDR. Thus, greater precision and accuracy is achieved based on the size of the register size.

[0046] The foregoing description of various embodiments of the claimed subject matter has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, so as to enable others skilled in the relevant art to understand the claimed subject matter, its various embodiments, and various modifications as may be suitable for the particular application contemplated.

Claims

1. 1. A method for converting data stored in a memory from a first format to a second format for machine learning (ML) operations, comprising: expanding the number of bits of the data stored in a double data rate (DDR) memory by one bit to form expanded data; determining whether the data stored in the DDR memory is signed or unsigned data; in response to determining that the data is signed, adding a sign value to a most significant bit of the extended data and copying the data to a least significant bit of the extended data; in response to determining that the data is unsigned, copying the data to less significant bits of the extended data and setting the most significant bit to an unsigned value; storing the augmented data in an on-chip memory (OCM) of a processing tile of a machine learning computer array; A method for providing the above.

2. The method of claim 1 , wherein the data is an unsigned integer.

3. The method of claim 1 , wherein the data is a signed integer.

4. The method of claim 1 , wherein the data is 8 bits and the extended data is 9 bits.

5. The method of claim 1 , wherein the extended data is int9 data.

6. tracking whether the data stored in the DDR memory is signed or unsigned; scheduling an appropriate instruction for the expanded data based on whether the data is signed or unsigned; The method of claim 1 , further comprising:

7. 7. The method of claim 6, further comprising performing an arithmetic logic unit (ALU) operation on the expanded data as an operand.

8. The method of claim 7 , further comprising storing results of the operations in the OCM of the processing tile of the machine learning computer array.

9. The method of claim 8 , further comprising storing the results stored in the OCM in the DDR memory.

10. 10. The method of claim 9, further comprising, prior to storing the result in the DDR memory, adjusting the value of the result to a maximum value of a range for the data if the value of the result exceeds the maximum value of the range for the data, and adjusting the value of the result to a minimum value of the range for the data if the value of the result is lower than a minimum value of the range for the data.

11. 11. The method of claim 9 or 10, further comprising the step of dropping the most significant bits of the result prior to storing the result from the OCM to the DDR memory.

12. 12. The method of claim 1, wherein the data stored in the DDR memory is an integer representation of the data for floating point data.

13. 13. The method of claim 12, wherein the floating point data is scaled and quantized to form the data in the first format.

14. 14. The method of claim 12 or 13, wherein a first scaling value is used to convert the floating-point data to an int8 format and a second scaling value is used to convert the floating-point data to a uint8 format.

15. a double data rate (DDR) memory configured to store integer data in a first format; A machine learning processing unit having a plurality of processing tiles; Each processing tile comprises: A system having an on-chip memory (OCM) configured to accept and maintain extended data converted from the integer data in the first format from the DDR memory for various ML operations, the extended data including one additional bit compared to the integer data in the first format, and if the integer data in the first format is signed, the most significant bit of the extended data is signed, and if the integer data in the first format is unsigned, the most significant bit of the extended data is set to an unsigned value, and the least significant bit of the extended data is the same as the integer data in the first format.

16. 16. The system of claim 15, wherein the integer data in the first format is either an int8 or a uint8.

17. 17. The system of claim 15 or 16, wherein the expanded data is int9.

18. 18. The system of claim 15, wherein whether the integer data in the first format stored in the DDR memory is signed or unsigned is tracked, and appropriate instructions are scheduled depending on whether the integer data in the first format is signed or unsigned.

19. 19. The system of claim 15, wherein the expanded data is an operand for an operation.

20. The system of claim 19 , wherein a result of the operation is stored in the OCM.

21. 21. The system of claim 20, wherein the results of the operations stored in the OCM are further stored in the DDR memory.

22. 22. The system of claim 21, wherein the value of the result is adjusted to the maximum value of the range for the integer data in the first format if the value of the result exceeds a maximum value, and the value of the result is adjusted to the minimum value of the range for the integer data in the first format if the value of the result is lower than a minimum value of the range for the data before storing the result in the DDR memory.

23. 23. The system of claim 21 or 22, wherein the most significant bits of the result are dropped before storing the result in the DDR memory.

24. 24. The system of claim 15, wherein the integer data in the first format is an integer representation of floating point data, and the floating point data is scaled and quantized to form the integer data in the first format.

25. 25. The system of claim 24, wherein a first scaling value is used to convert the floating point data to an int8 format and a second scaling value is used to convert the floating point data to a uint8 format.

Images