Method for decomposing high-precision matrix multiplication into multiple matrix multiplications of different data types

Abstract

In some examples, a method of performing matrix multiplication using mixed precision decomposition of operands includes converting a first matrix including a plurality of elements having a first precision to a second matrix, where the second matrix includes a plurality of elements having a second precision, the second precision being lower than the first precision; generating a first output matrix comprising a product of the second matrix and a third matrix, where the third matrix comprises a plurality of elements having the second precision, the elements of the first output matrix being stored at the first precision or converted to the first precision; calculating a residual value matrix using the first matrix and the second matrix, where the residual value matrix comprises elements representing respective differences between elements of the first matrix and elements of the second matrix; the elements of the residual matrix are scaled using a first scaling factor to generate a first scaled matrix.

Description

Technical Field

[0001] The present invention generally relates to matrix multiplication. Various aspects are directed to decomposing a high-precision matrix multiplication into multiple matrix multiplications of different data types. Background Art

[0002] In the field of artificial intelligence (AI), multiple computations are required to complete AI workloads. AI accelerators include high-performance parallel computing machines designed specifically to efficiently process AI workloads (such as neural networks).

[0003] AI accelerators are optimized primarily for reducing the precision of operands, such as using half-precision binary floating point formats, truncated single-precision 32-bit floating point formats (e.g., bfloat), or integer formats (e.g., signed integers such as int8, int4, etc.). AI accelerators typically provide a tensor processing unit (TPU), which includes specialized dedicated hardware such as tensor cores / cubes for such operands, or some form of vector processor for performing a large number of low-precision calculations (e.g., down to 8-bit precision). However, such devices typically need to fall back to significantly slower processing units for higher-precision operations.

[0004] Some AI-related workloads require higher numerical precision and may result in degraded results (e.g., lower click-through rates for recommendations) when deployed with lower precision (e.g., float16), which cannot be overcome by simple solutions such as scaling or shifting the value range of intermediate results and / or training / constant model parameters (e.g., weight matrices).

[0005] Deep neural network (DNN) training scenarios may also require higher numerical precision, especially for gradients and learning parameters, to converge to the highest achievable result quality (e.g., classification accuracy) using similar computation / training iterations. In addition to AI / DNN related applications, high performance computing (HPC) applications (such as solvers, simulations, etc.) are often run using double-precision operations to meet target quality requirements, which effectively limits access to many current commercial high-performance energy-efficient AI accelerators, or requires running these accelerators in higher precision modes, but at the cost of significantly reduced throughput and higher power consumption. Summary of the invention

[0006] The object of the present invention is to provide an efficient method of performing higher precision operations in a device that can be implemented in software or with minimal hardware modifications.

[0007] These and other objects are achieved by the features of the independent claims.

[0008] Further implementations are apparent from the dependent claims, the description and the drawings.

[0009] A first aspect of the present invention provides a method for performing matrix multiplication using mixed precision decomposition of operands, the method comprising: converting a first matrix including multiple elements with a first precision into a second matrix, wherein the second matrix includes multiple elements with a second precision, and the second precision is lower than the first precision; generating a first output matrix including the product of the second matrix and a third matrix, wherein the third matrix includes multiple elements with the second precision, and the elements of the first output matrix are stored or converted to the first precision with the first precision; calculating a residual value matrix using the first matrix and the second matrix, wherein the residual value matrix includes elements representing corresponding differences between elements of the first matrix and elements of the second matrix; scaling the residual value matrix using a first scaling factor; elements to generate a first scaling matrix, wherein the first scaling factor is determined based on the absolute values ​​of the elements of the residual value matrix; a first intermediate matrix is ​​generated by calculating the element-by-element product of the first scaling matrix using a fourth matrix, wherein the fourth matrix is ​​determined by using a version of the third matrix with the first precision scaled by a second scaling factor, wherein the second scaling factor is determined based on the absolute values ​​of the elements of the third matrix with the first precision; a fifth matrix is ​​generated according to the element-by-element quotient of the first intermediate matrix and the first scaling factor; a product of the fifth matrix and the reciprocal of the second scaling factor is calculated to generate a second output matrix with the first precision; and a final output matrix with the first precision is generated using the first output matrix and the second output matrix.

[0010] Therefore, a high-precision (e.g., float32) matrix multiplication can be decomposed into multiple matrix multiplications of different data types (e.g., float16, int8, etc.). Multiple lower-precision data types allow for the use of more energy-efficient or faster computational units. For example, some hardware can perform float16 operations 64 times faster than float32 operations, and int8 operations can be performed 128 times faster than float32 operations. This has a direct beneficial impact on the memory bandwidth and buffer size required for efficient processing, which is proportional to the data type selected for each matrix multiplication operation.

[0011] In an implementation of the first aspect, the method may further include: generating a second intermediate matrix by calculating the element-by-element product of a fifth matrix and a sixth matrix, wherein the fifth matrix is ​​determined by using a version of the first matrix scaled by a third scaling factor, and the third scaling factor is determined based on the maximum value of the absolute values ​​of the elements of the first matrix, and the sixth matrix includes: elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix, wherein the elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix are scaled using a fourth scaling factor, and the fourth scaling factor is determined based on the absolute value of the difference between the elements of the third matrix with the first precision and the elements of the third matrix.

[0012] In one example, the method may further include: determining an element-by-element quotient of the second intermediate matrix and the third scaling factor to generate a third intermediate matrix; calculating the product of the third intermediate matrix and the inverse of the fourth scaling factor to generate a third output matrix with the first precision; using the first output matrix, the second output matrix and the third output matrix to generate a final output matrix with the first precision. The first precision may include at least one of the following: single-precision floating point format precision, double-precision floating point format precision, or quad-precision floating point format precision; and / or the second precision may include at least one of the following: single-precision floating point format precision or half-precision floating point format precision; and / or the third precision may include at least one of the following: half-precision floating point format precision or integer format precision.

[0013] The method may further include: pre-scaling the first matrix and / or the second matrix using a predetermined fixed scaling factor or using a scaling factor determined based on the value distribution of the elements of the first matrix and / or the second matrix. The method may further include: determining a first maximum value from the modulus of the elements of the residual value matrix; generating the first scaling factor using the first maximum value. The method may further include: determining a second maximum value from the modulus of the elements of the second matrix having the first precision; generating the second scaling factor using the second maximum value.

[0014] A second aspect of the present invention provides a device for performing matrix multiplication using mixed precision decomposition of operands, the device comprising a processor and a memory coupled to the processor, wherein the memory comprises instructions, which, when executed by the processor, cause the device to perform the following steps: converting a first matrix comprising multiple elements having a first precision into a second matrix, wherein the second matrix comprises multiple elements having a second precision, and the second precision is lower than the first precision; generating a first output matrix comprising the product of the second matrix and a third matrix, wherein the third matrix comprises multiple elements having the second precision, and the elements of the first output matrix are stored or converted to the first precision with the first precision; using the first matrix and the second matrix to calculate a residual value matrix, wherein the residual value matrix comprises a matrix representing the difference between the elements of the first matrix and the elements of the second matrix; elements of the corresponding difference values; scaling the elements of the residual value matrix using a first scaling factor to generate a first scaling matrix, wherein the first scaling factor is determined based on the absolute values ​​of the elements of the residual value matrix; generating a first intermediate matrix by calculating the element-by-element product of the first scaling matrix using a fourth matrix, wherein the fourth matrix is ​​determined by using a version of the third matrix with the first precision scaled by a second scaling factor, wherein the second scaling factor is determined based on the absolute values ​​of the elements of the third matrix with the first precision; generating a fifth matrix according to the element-by-element quotient of the first intermediate matrix and the first scaling factor; calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix with the first precision; and generating a final output matrix with the first precision using the first output matrix and the second output matrix.

[0015] In an implementation of the second aspect, the memory may also include instructions, which, when executed by the processor, cause the device to perform the following steps: generate a second intermediate matrix by calculating the element-by-element product of a fifth matrix and a sixth matrix, wherein the fifth matrix is ​​determined by using a version of the first matrix scaled by a third scaling factor, and the third scaling factor is determined based on the maximum value of the absolute values ​​of the elements of the first matrix, and the sixth matrix includes: elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix, wherein the elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix are scaled using a fourth scaling factor, and the fourth scaling factor is determined based on the absolute value of the difference between the elements of the third matrix with the first precision and the elements of the third matrix. The memory may also include instructions, which, when executed by the processor, cause the device to perform the following steps: determine the element-by-element quotient of the second intermediate matrix and the third scaling factor to generate a third intermediate matrix; calculate the product of the third intermediate matrix and the inverse of the fourth scaling factor to generate a third output matrix with the first precision; use the first output matrix, the second output matrix, and the third output matrix to generate a final output matrix with the first precision. The memory may also include instructions, which, when executed by the processor, cause the device to perform the following steps: pre-scale the first matrix and / or the second matrix using a predetermined fixed scaling factor or using a scaling factor determined based on the value distribution of the elements of the first matrix and / or the second matrix.

[0016] A third aspect of the present invention provides a machine-readable storage medium, the machine-readable storage medium comprising instructions for performing matrix multiplication using mixed precision decomposition of operands, the instructions being executable by a processor of a device so that the device performs the following steps: converting a first matrix comprising multiple elements having a first precision into a second matrix, wherein the second matrix comprises multiple elements having a second precision, and the second precision is lower than the first precision; generating a first output matrix comprising the product of the second matrix and a third matrix, wherein the third matrix comprises multiple elements having the second precision, and the elements of the first output matrix are stored or converted to the first precision with the first precision; and calculating a residual value matrix using the first matrix and the second matrix, wherein the residual value matrix comprises corresponding difference values ​​between elements representing the first matrix and elements representing the second matrix. elements of the matrix; scaling the elements of the residual value matrix using a first scaling factor to generate a first scaling matrix, wherein the first scaling factor is determined based on the absolute values ​​of the elements of the residual value matrix; generating a first intermediate matrix by calculating the element-by-element product of the first scaling matrix using a fourth matrix, wherein the fourth matrix is ​​determined by using a version of the third matrix with the first precision scaled by a second scaling factor, wherein the second scaling factor is determined based on the absolute values ​​of the elements of the third matrix with the first precision; generating a fifth matrix according to the element-by-element quotient of the first intermediate matrix and the first scaling factor; calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix with the first precision; and generating a final output matrix with the first precision using the first output matrix and the second output matrix.

[0017] In an implementation of the third aspect, the machine-readable storage medium may also include instructions, which can be executed by the processor of the device, so that the device performs the following steps: generating a second intermediate matrix by calculating the element-by-element product of a fifth matrix and a sixth matrix, wherein the fifth matrix is ​​determined by using a version of the first matrix scaled by a third scaling factor, and the third scaling factor is determined based on the maximum value of the absolute values ​​of the elements of the first matrix, and the sixth matrix includes: elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix, wherein the elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix are scaled using a fourth scaling factor, and the fourth scaling factor is determined based on the absolute value of the difference between the elements of the third matrix with the first precision and the elements of the third matrix. The machine-readable storage medium may also include instructions that can be executed by the processor of the device, so that the device performs the following steps: determining the element-by-element quotient of the second intermediate matrix and the third scaling factor to generate a third intermediate matrix; calculating the product of the third intermediate matrix and the inverse of the fourth scaling factor to generate a third output matrix with the first precision; using the first output matrix, the second output matrix and the third output matrix to generate a final output matrix with the first precision. The machine-readable storage medium may also include instructions that can be executed by the processor of the device, so that the device performs the following steps: pre-scaling the first matrix and / or the second matrix using a predetermined fixed scaling factor or using a scaling factor determined based on the value distribution of the elements of the first matrix and / or the second matrix.

[0018] These and other aspects of the invention will be apparent from and elucidated with reference to one or more of the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to make the present invention more easily understood, embodiments are now described by way of example with reference to the accompanying drawings, in which:

[0020] Figure 1 is a schematic diagram of a method for performing matrix multiplication using mixed precision decomposition of operands according to an example;

[0021] Figure 2 is a schematic diagram of a machine according to an example;

[0022] Figure 3 is a flow chart of a method for performing matrix multiplication using mixed-precision decomposition of operands according to one example. DETAILED DESCRIPTION

[0023] The exemplary embodiments are described below in sufficient detail to enable one of ordinary skill in the art to implement and practice the systems and processes described herein.It is important to understand that the embodiments may be provided in many alternative forms and should not be construed as limited to only the examples described herein.

[0024] Therefore, although the embodiments may be modified in various ways and take various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. It is not intended to be limited to the specific forms disclosed. On the contrary, all modifications, equivalents and alternatives within the scope of the appended claims should be included. In all drawings and appropriate specific embodiments, the elements of the exemplary embodiments are consistently represented by the same reference numerals.

[0025] The terms used to describe the embodiments herein are not intended to limit the scope. The articles "one" and "the" are singular forms because they have only one referent, but the use of the singular form herein should not exclude the presence of multiple referents. In other words, unless the context otherwise clearly indicates, the elements mentioned in the singular form can be one or more in number. It should be further understood that the term "comprising" used herein is used to illustrate the presence of the features, items, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, items, steps, operations, elements, parts and / or their combinations. The term "and / or" is merely an association relationship describing an associated object, indicating that three relationships can exist, so that A and / or B can represent that A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " generally indicates that the associated object is in an "or" relationship.

[0026] Unless otherwise defined, all terms (including technical and scientific terms) used in this article should be interpreted as customary usage in the art. It should also be understood that unless explicitly defined in this article, terms in common usage should also be interpreted as customary usage in the relevant art, and should not be interpreted as idealized or overly formal meanings.

[0027] The following includes specific information related to the implementation of the present invention. The accompanying drawings and the detailed disclosure thereof are only for the implementation. However, the present invention is not limited to these implementations. Other variations and implementations of the present invention will be apparent to those skilled in the art.

[0028] The phrases "in one implementation" or "in some implementations" may each refer to one or more of the same or different implementations. The term "coupled" is defined as connected directly or indirectly through intermediate components, and is not necessarily limited to physical connections. The expression "at least one of A, B, and C" or "at least one of: A, B, and C" means "only A, or only B, or only C, or any combination of A, B, and C."

[0029] The terms "system" and "network" are used interchangeably.

[0030] For the purpose of explanation and non-limitation, specific details such as functional entities, technologies, protocols and standards are described to provide an understanding of the present invention. In other examples, detailed disclosure of well-known methods, technologies, systems and architectures is omitted to avoid obscuring the present invention with unnecessary details.

[0031] Those skilled in the art will appreciate that any disclosed network function or algorithm may be implemented by hardware, software, or a combination of software and hardware. The disclosed functions may correspond to modules, which may be software, hardware, firmware, or any combination thereof.

[0032] Software implementation may include machine and / or computer readable and / or executable instructions stored in a machine and / or computer readable medium such as a memory or other type of storage device. One or more microprocessors or general-purpose computers with communication processing capabilities may be programmed with corresponding executable instructions and execute the disclosed network functions or algorithms.

[0033] The microprocessor or general purpose computer may include an Applications Specific Integrated Circuitry (ASIC), a programmable logic array and / or use one or more digital signal processors (DSP). Although some of the disclosed implementations are directed to software installed and executed on computer hardware, alternative implementations implemented as firmware or hardware or as a combination of hardware and software are fully within the scope of the present invention. Computer readable media include, but are not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), cassette, magnetic tape, disk storage, or any other equivalent medium capable of storing computer readable instructions.

[0034] According to one example, a high-precision matrix multiplication is performed using a lower-precision computational unit. That is, a high-precision matrix multiplication can be decomposed into multiple lower-precision matrix multiplications (MatMul) of different data types. Values ​​can be scaled according to input data, and some operations can be performed with lower precision (e.g., full-precision format is float32, reduced-precision format is float16, and strong reduced-precision format is int8).

[0035] This can further reduce the computational workload, thereby achieving higher throughput.

[0036] According to one example, a high-precision matrix multiplication (data type "HP") can be decomposed into two or more lower-precision matrix multiplication data types (data types "LP1", "LP2"), and the result can be stored in an accumulation data type ("AP"). Therefore, the high-precision matrix can be converted to LP1 type and a normal MatMul can be performed, thereby keeping the result in AP type. Then, a quantized residual is calculated using the first input matrix, and in one example, the residual includes the difference between its HP and LP1 representations. The quantized residual is then stored back in HP. The quantized residual is scaled based on the value distribution of the quantized residual (e.g., the maximum absolute value in the matrix) to match a range of values ​​suitable for representation in LP2 and converted to a selected data type (e.g., LP2). The second input matrix (not its residual) is converted to LP2 type after being scaled in HP using the scaling factor of the second input matrix. The two matrices are multiplied, the result is converted to AP type, and the result is scaled using the inverse of the product of the two scaling factors. The resulting matrix is ​​added to the MatMul result of the previously calculated LP1 type matrix. The residual calculation can be repeated with the swapped input matrices or added to the final result.

[0037] The method of performing matrix multiplication using mixed precision decomposition of operands can be implemented on an AI accelerator device or GPU, which has, for example, significantly different throughputs for calculating matrix multiplication of different data types (e.g., Huawei's Ascend device, or Nvidia and AMD's GPUs). Typically, such accelerator devices have memory / RAM to store / load inputs and outputs, as well as potential intermediate results. One application scenario for such devices is to run AI models on such devices. These models include a series of processing steps, for which the numerical precision of float16 or int8 is usually sufficient, and these models can be run on high-throughput matrix multiplication units available for these data types. Certain processing steps in some models require higher precision. This can be challenging because either no unit can perform the calculation with sufficient precision or the throughput of the unit is very low. In this case, the present method can be applied to replace high-precision matrix multiplication with multiple operations with high-throughput, low-precision units.

[0038] Figure 1 is a diagram of a method for performing matrix multiplication using mixed-precision decomposition of operands. Figure 1 In the example of , a first matrix 101 (including operands) having a first precision is provided. The matrix 101 is converted 105 to a second matrix 104 having a second precision, wherein the second precision is a lower precision format version (e.g., float16) than the first precision (e.g., float32). Thus, a matrix x is provided fp16 (104). A third matrix w is provided fp16 (103). The third matrix 103 may be a lower precision version of the matrix 102. For example, the matrix 102 may be provided with a first precision, and the matrix 103 may be provided with a second precision.

[0039] The product 109 of the second matrix 104 and the third matrix 103 is calculated and the result is converted 111 or stored in a higher precision format (e.g., float32) 113 (referred to herein as ), for example, first precision.

[0040] According to an example, the residual R can be calculated X =x–x fp16 and R W =w–w fp16 (As shown in 115). For x residual R X , the second matrix 104 (forming a lower precision version of x, i.e., x) is subtracted 119 from the first matrix 101 (i.e., the full precision or higher precision version) fp16 ). A first scaling factor α is also used X120 scaling x residual R X In one example, α x Can be a fixed scaling factor or based on R X The calculated scaling factor. Scaling factor α X 120 includes values ​​determined based on the absolute values ​​of the elements of the residual value matrix. Figure 1 In the example, the (scalar) scaling factor α X 120 is determined based on the maximum value of the elements of the residual matrix Rx 119 and a factor (here 127). Figure 1 In the example, the value 127 is used to match the value range of the int8 data type.

[0041] The elements of the residual value matrix 119 are scaled using the first scaling factor 120 to generate a first scaling matrix 124 (eg Figure 1 int8 in the example), also called matrix Q XX That is, reference Figure 1 , the matrix Q XX 124 is calculated from the element-wise product of Rx 119 and scaling factor 120.

[0042] The first intermediate matrix 127 is generated by calculating the element-wise product of the first scaling matrix 124 using the fourth matrix 130. The fourth matrix 130 is determined using the matrix 102 scaled by the second scaling factor 125 (ie, a higher precision version of the third matrix 103). X 125 is determined based on the absolute values ​​of the elements of the matrix 102. Figure 1 In the example, the second scaling factor β x 125 is determined based on the maximum value of the elements of the second matrix having the first precision 102 and the factor (here 127). x refers to the residual in x, but x can be applied to the matrix w.

[0043] A fifth matrix 132 is generated according to the element-by-element quotient of the first intermediate matrix 127 and the first scaling factor 120. The product of the fifth matrix 132 and the inverse of the second scaling factor 125 is calculated to generate a second output matrix 129 having the first precision. That is, the fifth matrix 132 is then scaled using the inverse of the second scaling factor 128, and the fifth matrix 132 is quantized to the first precision (i.e., a higher precision format) (e.g., float32) to obtain a second output matrix 129.

[0044] A similar process (as described in 117) may be followed to determine the residual associated with matrix 102. For w residual B w , use the scaling factor 131 to scale the matrix 101 (i.e. x fp32). In one example, the scaling factor 131 comprises the inverse of the maximum value of the modulus of the elements of the first matrix 101 having the first precision. A numerical value (here 127) is provided as part of the scaling factor 131, which is based on the desired precision of the scaling result (such as 120 at 115). That is, in Figure 1 In the example of , a lower precision format (e.g. Figure 1 The fifth matrix 133 of int8 in the example.

[0045] Matrix multiplication is performed using the fifth matrix 133 and the matrix 135 in a lower precision format (also referred to as the sixth matrix) to generate a second intermediate matrix 150. Figure 1 In the example of , a sixth matrix 135 of lower precision is calculated from the quotient of the difference between the values ​​of the elements in matrix 102 and matrix 103, and the maximum value determined from the modulus of the difference between the values ​​of the elements in matrix 102 and matrix 103. As previously described, a numerical factor (here 127) may be applied.

[0046] A third intermediate matrix 137 is generated from the element-by-element quotient of the second intermediate matrix 150 and the scaling factor 131. The third intermediate matrix 137 is then scaled using the inverse of the scaling factor 138 described above with reference to the matrix 135 to produce a third output matrix 139 ( ). The final output matrix with first precision is and (Optional) Sum.

[0047] According to an example, a combination of different data types, i.e., first precision, second precision, third precision, etc., may be used to represent a higher precision format and / or a lower precision format, etc. For example, the first precision may include at least one of the following: single-precision floating point format precision, double-precision floating point format precision, or quad-precision floating point format precision; and / or the second precision may include at least one of the following: single-precision floating point format precision or half-precision floating point format precision; and / or the third precision may include at least one of the following: half-precision floating point format precision or integer format precision. A value and / or result and / or matrix (and / or any one or more elements thereof) having a higher precision format refers to a precision format that can store data with higher accuracy and / or a precision format with a higher numerical dynamic range compared to a lower precision format. For example, float32 is a higher precision format compared to float16, int8, and int32, while float32 is a lower precision format compared to float64, and so on.

[0048] In one implementation, the reduced precision data type described above may be used to store results representing noise measurements, which may be stored in the same data format as higher precision operations, whereby lower precision operations may be calculated using an analog matrix multiplication device, higher precision operations may be performed using digital circuits or higher precision modes of the analog matrix multiplication device, and so on.

[0049] According to one example, the additional residual MatMul (e.g. R W ·R X ) or the residuals of the residuals (e.g. using Item) to expand the above reference Figure 1 Furthermore, some partial results / steps can be precomputed or reused in multiple MatMul operations (e.g. only all values ​​and matrices that depend on W). Similarly, the method can be applied independently to tiles of the matrix, possibly reusing partial results on different tiles.

[0050] According to an example, the first and / or second and / or third scaling factors, etc. may include a fixed value, a current maximum absolute value, a historical average of the maximum absolute value, etc. The value range of the input matrix may be first pre-scaled based on a fixed factor or based on a value distribution to avoid overflow, and the result may be reversely scaled after applying the above method.

[0051] Examples of the present invention may be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware, etc. The machine-readable instructions may be included in a computer-readable storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) having a computer-readable program code therein or thereon.

[0052] The present invention is described with reference to the flow chart and / or block diagram of the method, device and system according to the example of the present invention. Although the flow chart described above shows a specific execution order, the execution order may be different from that described. The frame associated with a flow chart can be combined with the frame of another flow chart. In some examples, some frames of the flow chart may be unnecessary, and / or additional frames can be added. It should be understood that each flow process and / or frame in the flow chart and / or block diagram, and the combination of the flow process and / or figure in the flow chart and / or block diagram can be implemented by the above-mentioned machine-readable instructions.

[0053] For example, machine-readable instructions can be executed by a machine such as a general-purpose computer, a user device including a smart device (e.g., a processor of a smart phone, a dedicated computer, an embedded processor, or other programmable data processing device) and other platforms to implement the functions described in the specification and the figure. In particular, a processor or a processing device can execute machine-readable instructions. Therefore, the module of the device (e.g., a module implementing a converter, which is used to convert one precision format to another precision format (such as float32 to float16, etc.), and / or a module for performing matrix multiplication, and / or a module for applying a scaling factor to a matrix) can be implemented by a processor that executes machine-readable instructions stored in a memory, or by a processor that runs according to instructions embedded in a logic circuit. The term "processor" can be broadly interpreted to include a CPU, a processing unit, an ASIC, a logic unit, or a programmable gate group, etc. These methods and modules can be executed by a single processor or by multiple processors.

[0054] The machine-readable instructions may also be stored in a computer-readable memory, which may direct a computer or other programmable data processing device to operate in a specific mode. For example, the instructions may be provided in a non-transitory computer-readable storage medium encoded with instructions, which may be executed by a processor.

[0055] Figure 2 2 is a schematic diagram of a machine according to an example. The machine 200 may be a system or apparatus, a user device or a portion thereof, or a low-level lightweight state machine within an AI accelerator or processor, etc. The machine 200 includes a processor 203 and a memory 205, the memory 205 being used to store instructions 202 executable by the processor 203. The machine includes a storage device 209, the storage device 209 being used to store data 201, the data 201 representing the values ​​of matrices and / or elements of matrices, scaling factors, etc., such as those described above with reference to Figure 1 described.

[0056] The instructions 207 executable by the processor 203 may cause the machine 200 to: convert a first matrix including a plurality of elements having a first precision into a second matrix, wherein the second matrix includes a plurality of elements having a second precision, the second precision being lower than the first precision; generate a first output matrix including a product of the second matrix and a third matrix, wherein the third matrix includes a plurality of elements having the second precision, and the elements of the first output matrix are stored or converted to the first precision; calculate a residual value matrix using the first matrix and the second matrix, wherein the residual value matrix includes elements representing corresponding differences between elements of the first matrix and elements of the second matrix; scale the elements of the residual value matrix using a first scaling factor to generate A first scaling matrix, wherein the first scaling factor is determined based on the absolute values ​​of the elements of the residual value matrix; generating a first intermediate matrix by calculating the element-by-element product of the first scaling matrix using a fourth matrix, wherein the fourth matrix is ​​determined using a version of the third matrix with the first precision that has been scaled using a second scaling factor, and the second scaling factor is determined based on the absolute values ​​of the elements of the third matrix with the first precision; generating a fifth matrix based on the element-by-element quotient of the first intermediate matrix and the first scaling factor; calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix with the first precision; and generating a final output matrix with the first precision using the first output matrix and the second output matrix.

[0057] Thus, machine 200 may implement a method for performing matrix multiplication using mixed-precision decomposition of operands.

[0058] The machine-readable instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable data processing device performs a series of operations to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide operations for implementing the functions specified by the processes in the flowchart and / or the boxes in the block diagram.

[0059] Furthermore, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, in which the computer software or product is stored and includes a plurality of instructions, such as machine-readable instructions, that cause a computer device to implement the method described in the examples of the present invention.

[0060] In some examples, some methods may be performed in a cloud-based or web-based environment. A cloud computing environment may provide various services and applications over the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessed through a web browser or other remote interface of a user device. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

[0061] Figure 31 is a schematic diagram of a method for performing matrix multiplication using mixed precision decomposition of operands according to an example. In block 301, a first matrix 101 including a plurality of elements having a first precision is converted to a second matrix 104, wherein the second matrix 104 includes a plurality of elements having a second precision, the second precision being lower than the first precision. In an example, a matrix 102 including a plurality of elements having a first precision may be converted to a third matrix 103, wherein the third matrix 103 includes a plurality of elements having a second precision, the second precision being lower than the first precision.

[0062] In block 303, a first output matrix is ​​generated comprising the product of the second matrix 104 and the third matrix 103. The third matrix 103 comprises a plurality of elements having the second precision, and the elements of the first output matrix are stored or converted 111 to the first precision in order to provide a first output matrix 113 having the first precision.

[0063] In block 305, a residual value matrix Rx 119 is generated using the first matrix 101 and the second matrix 104. The residual value matrix includes elements representing corresponding differences between elements of the first matrix 101 and elements of the second matrix 104.

[0064] In block 307, the elements of the residual value matrix are scaled using the first scaling factor 120 to generate a first scaling matrix 124, wherein the first scaling factor 120 is determined based on the absolute values ​​of the elements of the residual value matrix. Thus, the elements of the first scaling matrix 124 are provided with a third precision by the scaling factor 120. In one example, the third precision is a lower precision than the second precision or the same precision as the second precision. In one example, the first scaling matrix 124 may include a matrix 130 (e.g., int8) in a lower precision format.

[0065] In box 309, the first intermediate matrix 127 is generated by calculating the element-by-element product of the first scaling matrix 124 using the fourth matrix 130, wherein the fourth matrix 130 is determined using a version of the third matrix with the first precision that has been scaled using the second scaling factor 125, and the second scaling factor 125 is determined based on the absolute values ​​of the elements of the third matrix with the first precision.

[0066] In block 311 , a fifth matrix 132 is generated from the element-by-element quotient of the first intermediate matrix 127 and the scaling factor 120 .

[0067] In block 313 , the product 129 of the fifth matrix 132 and the inverse 128 of the second scaling factor 125 is calculated to generate a second output matrix 129 having the first precision.

[0068] In block 315, a final output matrix 140 is generated with a first precision using the first output matrix 113 and the second output matrix 129. In one example, a single element of the final output matrix 140 includes a sum of a corresponding element of the first output matrix 113 and a corresponding element of the second output matrix 129.

[0069] According to an example, a final output matrix 140 having a first precision is generated using the first output matrix 113, the second output matrix 129, and the third output matrix 139. In an example, a single element of the final output matrix 140 includes the sum of the corresponding element of the first output matrix 113, the corresponding element of the second output matrix 129, and the corresponding element of the third output matrix 139.

[0070] Although various embodiments have been described and / or shown herein in the context of a fully functional computing system, one or more of these exemplary embodiments can be distributed as a program product in various forms, regardless of which specific type of computer-readable storage medium is used to actually perform the distribution. The embodiments disclosed herein can also be implemented using software modules that perform certain tasks. These software modules can include scripts, batch processing, or other executable files that can be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules can be used to configure a computing system to perform one or more exemplary embodiments disclosed herein. In addition, one or more modules described herein can convert data, physical devices, and / or representations of physical devices from one form to another.

[0071] The above description is provided to enable those skilled in the art to better utilize the various aspects of the exemplary embodiments disclosed herein. The exemplary description is not exhaustive and is not limited to any precise form disclosed. Various modifications and changes may be made without departing from the spirit and scope of the present invention. The embodiments disclosed herein should be considered illustrative and not restrictive in all aspects. When determining the scope of the present invention, reference should be made to the attached claims and their equivalents.

Claims

1. A method for performing matrix multiplication using mixed-precision decomposition of operands, It is characterized in that The method comprises: converting a first matrix including a plurality of elements having a first precision into a second matrix, wherein the second matrix includes a plurality of elements having a second precision, the second precision being lower than the first precision; generating a first output matrix comprising a product of the second matrix and a third matrix, wherein the third matrix comprises a plurality of elements having the second precision, and elements of the first output matrix are stored in or converted to the first precision; Calculating a residual value matrix using the first matrix and the second matrix, wherein the residual value matrix includes elements representing corresponding differences between elements of the first matrix and elements of the second matrix; scaling the elements of the residual value matrix using a first scaling factor to generate a first scaling matrix, wherein the first scaling factor is determined based on absolute values ​​of the elements of the residual value matrix; generating a first intermediate matrix by computing an element-wise product of the first scaled matrix using a fourth matrix, wherein the fourth matrix is ​​determined by using a version of the third matrix having the first precision scaled by a second scaling factor, the second scaling factor being determined based on absolute values ​​of elements of the third matrix having the first precision; generating a fifth matrix according to an element-by-element quotient of the first intermediate matrix and the first scaling factor; Calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix having the first precision; A final output matrix having the first precision is generated using the first output matrix and the second output matrix.

2. The method according to claim 1, It is characterized in that The method further comprises: A second intermediate matrix 150 is generated by calculating the element-by-element product of a fifth matrix 133 and a sixth matrix 135, wherein the fifth matrix is ​​determined by using a version of the first matrix scaled by a third scaling factor, the third scaling factor being determined based on the maximum value of the absolute values ​​of the elements of the first matrix, and the sixth matrix includes: elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix, wherein the elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix are scaled using a fourth scaling factor, the fourth scaling factor being determined based on the absolute value of the difference between the elements of the third matrix with the first precision and the elements of the third matrix.

3. The method according to claim 2, It is characterized in that The method further comprises: determining an element-by-element quotient of the second intermediate matrix and the third scaling factor to generate a third intermediate matrix; Calculating a product of the third intermediate matrix and the inverse of the fourth scaling factor to generate a third output matrix having the first precision; A final output matrix having the first precision is generated using the first output matrix, the second output matrix, and the third output matrix.

4. The method according to any one of the preceding claims, It is characterized in that The first precision includes at least one of the following: single-precision floating point format precision, double-precision floating point format precision or quad-precision floating point format precision; and / or the second precision includes at least one of the following: single-precision floating point format precision or half-precision floating point format precision; and / or the third precision includes at least one of the following: half-precision floating point format precision or integer format precision.

5. The method according to any one of the preceding claims, It is characterized in that The method further comprises: The first matrix and / or the second matrix are pre-scaled using a predetermined fixed scaling factor or using a scaling factor determined based on a distribution of values ​​of elements of the first matrix and / or the second matrix.

6. The method according to any one of the preceding claims, It is characterized in that The method further comprises: determining a first maximum value from the moduli of the elements of the residual value matrix; The first scaling factor is generated using the first maximum value.

7. The method according to any one of the preceding claims, It is characterized in that The method further comprises: determining a second maximum value from the moduli of the elements of the second matrix having the first precision; The second scaling factor is generated using the second maximum value.

8. An apparatus for performing matrix multiplication using mixed precision decomposition of operands, It is characterized in that The apparatus comprises a processor and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the apparatus to perform the following steps: converting a first matrix including a plurality of elements having a first precision into a second matrix, wherein the second matrix includes a plurality of elements having a second precision, the second precision being lower than the first precision; generating a first output matrix comprising a product of the second matrix and a third matrix, wherein the third matrix comprises a plurality of elements having the second precision, and elements of the first output matrix are stored in or converted to the first precision; Calculating a residual value matrix using the first matrix and the second matrix, wherein the residual value matrix includes elements representing corresponding differences between elements of the first matrix and elements of the second matrix; scaling the elements of the residual value matrix using a first scaling factor to generate a first scaling matrix, wherein the first scaling factor is determined based on absolute values ​​of the elements of the residual value matrix; generating a first intermediate matrix by computing an element-wise product of the first scaled matrix using a fourth matrix, wherein the fourth matrix is ​​determined by using a version of the third matrix having the first precision scaled by a second scaling factor, the second scaling factor being determined based on absolute values ​​of elements of the third matrix having the first precision; generating a fifth matrix according to an element-by-element quotient of the first intermediate matrix and the first scaling factor; Calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix having the first precision; A final output matrix having the first precision is generated using the first output matrix and the second output matrix.

9. The device according to claim 8, It is characterized in that The memory further includes instructions, which, when executed by the processor, cause the apparatus to perform the following steps: A second intermediate matrix 150 is generated by calculating the element-by-element product of a fifth matrix 133 and a sixth matrix 135, wherein the fifth matrix is ​​determined by using a version of the first matrix scaled by a third scaling factor, the third scaling factor being determined based on the maximum value of the absolute values ​​of the elements of the first matrix, and the sixth matrix includes: elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix, wherein the elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix are scaled using a fourth scaling factor, the fourth scaling factor being determined based on the absolute value of the difference between the elements of the third matrix with the first precision and the elements of the third matrix.

10. The device according to claim 9, It is characterized in that The memory further includes instructions, which, when executed by the processor, cause the apparatus to perform the following steps: determining an element-by-element quotient of the second intermediate matrix and the third scaling factor to generate a third intermediate matrix; Calculating a product of the third intermediate matrix and the inverse of the fourth scaling factor to generate a third output matrix having the first precision; A final output matrix having the first precision is generated using the first output matrix, the second output matrix, and the third output matrix.

11. The device according to any one of claims 8 to 10, It is characterized in that The memory further includes instructions, which, when executed by the processor, cause the apparatus to perform the following steps: The first matrix and / or the second matrix are pre-scaled using a predetermined fixed scaling factor or using a scaling factor determined based on a distribution of values ​​of elements of the first matrix and / or the second matrix.

12. A machine-readable storage medium, It is characterized in that The machine-readable storage medium includes instructions for performing matrix multiplication using mixed precision decomposition of operands, the instructions being executable by a processor of a device to cause the device to perform the following steps: converting a first matrix including a plurality of elements having a first precision into a second matrix, wherein the second matrix includes a plurality of elements having a second precision, the second precision being lower than the first precision; generating a first output matrix comprising a product of the second matrix and a third matrix, wherein the third matrix comprises a plurality of elements having the second precision, and elements of the first output matrix are stored in or converted to the first precision; Calculating a residual value matrix using the first matrix and the second matrix, wherein the residual value matrix includes elements representing corresponding differences between elements of the first matrix and elements of the second matrix; scaling the elements of the residual value matrix using a first scaling factor to generate a first scaling matrix, wherein the first scaling factor is determined based on absolute values ​​of the elements of the residual value matrix; generating a first intermediate matrix by computing an element-wise product of the first scaled matrix using a fourth matrix, wherein the fourth matrix is ​​determined by using a version of the third matrix having the first precision scaled by a second scaling factor, the second scaling factor being determined based on absolute values ​​of elements of the third matrix having the first precision; generating a fifth matrix according to an element-by-element quotient of the first intermediate matrix and the first scaling factor; Calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix having the first precision; A final output matrix having the first precision is generated using the first output matrix and the second output matrix.

13. The machine-readable storage medium according to claim 12, It is characterized in that The machine-readable storage medium further includes instructions, which are executable by the processor of the device to cause the device to perform the following steps: A second intermediate matrix 150 is generated by calculating the element-by-element product of a fifth matrix 133 and a sixth matrix 135, wherein the fifth matrix is ​​determined by using a version of the first matrix scaled by a third scaling factor, the third scaling factor being determined based on the maximum value of the absolute values ​​of the elements of the first matrix, and the sixth matrix includes: elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix, wherein the elements determined based on the element-by-element difference between the third matrix with the first precision and the third matrix are scaled using a fourth scaling factor, the fourth scaling factor being determined based on the absolute value of the difference between the elements of the third matrix with the first precision and the elements of the third matrix.

14. The machine-readable storage medium according to claim 12 or 13, It is characterized in that The machine-readable storage medium further includes instructions, which are executable by the processor of the device to cause the device to perform the following steps: determining an element-by-element quotient of the second intermediate matrix and the third scaling factor to generate a third intermediate matrix; Calculating a product of the third intermediate matrix and the inverse of the fourth scaling factor to generate a third output matrix having the first precision; A final output matrix having the first precision is generated using the first output matrix, the second output matrix, and the third output matrix.

15. The machine-readable storage medium according to any one of claims 12 to 14, It is characterized in that The machine-readable storage medium further includes instructions, which are executable by the processor of the device to cause the device to perform the following steps: The first matrix and / or the second matrix are pre-scaled using a predetermined fixed scaling factor or using a scaling factor determined based on a distribution of values ​​of elements of the first matrix and / or the second matrix.

Images