Weight matrix processing method and device, equipment, storage medium and product

Abstract

The invention discloses a weight matrix processing method and device, equipment, a storage medium and a product. The method comprises the steps that a target weight matrix is loaded; wherein the target weight matrix is a third-precision weight matrix which is arranged in a preset format and is formed by packaging a first-precision weight matrix; carrying out unpacking and inverse quantization processing on the target weight matrix to obtain a weight matrix with second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; and storing the weight matrix with the second precision to an on-chip storage unit so as to be loaded to a tensor core to execute matrix multiplication operation. According to the embodiment of the invention, the weight matrix can be compatible with the operation logic of tcore, so that the matrix multiplication operation can be smoothly completed in the tcore.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a weight matrix processing method, apparatus, device, storage medium, and product. Background Technology

[0002] In large inference models, weight matrices often consume significant amounts of GPU memory, while activation matrices have relatively lower memory usage but a greater impact on model accuracy. Therefore, it's common practice to maintain high precision in the activation matrix while quantizing the weight matrix into a low-precision matrix for matrix multiplication, thus achieving a balance between inference accuracy and performance. Figure 1 It uses a weight matrix with precision of int4 (4-bit Integer) and a high-precision activation matrix to perform matrix multiply and accumulate (mma) calculations, and outputs the results to the output matrix.

[0003] However, current tensor cores (tcore) typically do not directly support matrix multiplication operations of int4 (4-bit Integer). Therefore, there is an urgent need for a weight matrix processing method that allows the processed weight matrix to be compatible with the tcore's operational logic and to successfully perform matrix multiplication operations within it. Summary of the Invention

[0004] This application provides a weight matrix processing method, apparatus, device, storage medium, and product to solve the problem that tcore does not directly support int4 matrix multiplication operations in the prior art.

[0005] To achieve the above objectives, embodiments of this application provide a weight matrix processing method, including: Load the target weight matrix; wherein, the target weight matrix is: a third-precision weight matrix packaged from a first-precision weight matrix and arranged in a preset format; The target weight matrix is ​​unpacked and dequantized to obtain a weight matrix with a second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; The weight matrix of the second precision is stored in the on-chip storage unit for loading into the tensor core to perform matrix multiplication operations.

[0006] As an improvement to the above solution, the preset format includes: Each data point in the third precision weight matrix is ​​arranged according to the coordinate mapping rules of the second burst mode; wherein, the second burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst. Each data point in the third precision weight matrix is ​​arranged in the height dimension according to the coordinate mapping rules of the first burst mode; wherein, the first burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst.

[0007] As an improvement to the above scheme, the first burst mode is specifically used to limit the data types loaded and the amount of data loaded in a single burst. The loaded target weight matrix includes: The target weight matrix is ​​loaded into the register according to the first burst mode.

[0008] As an improvement to the above scheme, the second burst mode is used to limit the data type stored and the amount of data stored in a single burst. The step of storing the weight matrix of the second precision to the on-chip storage unit includes: The weight matrix of the second precision is stored in the on-chip storage unit according to the second burst mode.

[0009] As an improvement to the above scheme, the target weight matrix is ​​pre-constructed by the host and stored in an off-chip storage unit.

[0010] As an improvement to the above scheme, the step of unpacking and dequantizing the target weight matrix to obtain a weight matrix of second precision includes: The target weight matrix is ​​unpacked to obtain the weight matrix with the first precision. The weight matrix of the first precision is dequantized to obtain the weight matrix of the second precision and loaded into the register.

[0011] As an improvement to the above scheme, the first precision is a 4-bit signed integer, the second precision is a 16-bit floating-point number, and the third precision is a 32-bit single-precision floating-point number.

[0012] As an improvement to the above scheme, the first precision is a 4-bit signed integer, the second precision is an 8-bit signed integer, and the third precision is a 32-bit single-precision floating-point number.

[0013] To achieve the above objectives, embodiments of this application also provide a weight matrix processing apparatus, including: A loading module is used to load a target weight matrix; wherein the target weight matrix is ​​a third-precision weight matrix packaged from a first-precision weight matrix and arranged in a preset format; The processing module is used to unpack and dequantize the target weight matrix to obtain a weight matrix with a second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; The save module is used to save the weight matrix of the second precision to the on-chip storage unit for loading into the tensor core to perform matrix multiplication operations.

[0014] To achieve the above objectives, embodiments of this application also provide a weight matrix processing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the weight matrix processing method as described above when executing the computer program.

[0015] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the weight matrix processing method as described above.

[0016] To achieve the above objectives, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implements the weight matrix processing method as described above.

[0017] Compared with existing technologies, the weight matrix processing method, apparatus, device, storage medium, and product provided in this application embodiment load a target weight matrix; wherein the target weight matrix is: a weight matrix of first precision packaged into a weight matrix of third precision arranged in a preset format; the target weight matrix is ​​unpacked and dequantized to obtain a weight matrix of second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; the weight matrix of second precision is stored in an on-chip storage unit for loading into the tensor core to perform matrix multiplication operations. Therefore, this application embodiment, from packaging to unpacking and finally to dequantization of the first precision weight matrix, ensures that the dequantized second precision weight matrix is ​​compatible with the Tcore's operational logic, thus successfully completing matrix multiplication operations in Tcore. Furthermore, the target matrix is ​​arranged in a preset format, ensuring the smooth execution of subsequent unpacking and dequantization. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of matrix multiplication provided by existing technology; Figure 2This application implements the coordinate mapping rules for the burst mode; Figure 3 This is a flowchart of a weight matrix processing method provided in this application; Figure 4 This is a block diagram of a weight matrix processing system provided in this application. Figure 5 This is a schematic diagram of a packaging method provided in an embodiment of this application; Figure 6 This is another schematic diagram illustrating a packaging method provided in an embodiment of this application; Figure 7 This is a formatted layout diagram of a target weight matrix provided in an embodiment of this application; Figure 8 This is a structural block diagram of a weight matrix processing device provided in an embodiment of this application; Figure 9 This is a structural block diagram of a weight matrix processing device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] In the description of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0021] In this application description, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0022] In this application description, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." The term "and / or" means at least one of the connected objects, such as A and / or B, indicating three cases: including only A, only B, and both A and B. Unless otherwise stated, the term "multiple" means two or more.

[0023] It is worth noting that the embodiments of this application are applicable to various artificial intelligence chips, such as graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), etc. The embodiments of this application do not specifically limit them.

[0024] It is worth noting that burst mode limits the data types loaded or stored, as well as the amount of data loaded or stored in a single burst (i.e., the amount of data transmitted in a single burst). Its coordinate mapping rules are as follows: Figure 2 .

[0025] Figure 2 In this context, S4, S8, U8, BF16, FP32, int, and uint are the data types specified for loading or storing. Specifically: S4: Signed 4-bit (4-bit signed integer); S8: Signed 8-bit (8-bit signed integer); U8: Unsigned 8-bit (8-bit unsigned integer); BF16: Brain Floating Point 16-bit (16-bit floating-point number); FP32: Floating Point 32-bit (32-bit single-precision floating-point number); int: Integer (signed integer); uint: Unsigned Integer (unsigned integer).

[0026] The Get / Set data quantity limits the amount of data loaded or stored in a single burst mode. For example, Get / Set data quantity 1 (i.e., burst1) means that the amount of data loaded or stored in a burst is 1, and so on.

[0027] N / A indicates not supported. / / +X bytes represents the interval X bytes between the two coordinates. For example, from Figure 2 It can be seen that one burst mode is the int8 burst8 mode, which indicates that the data type loaded or stored is limited to int8, and the number of data loaded or stored in one burst is 8. According to the coordinate mapping rules of this burst mode, the corresponding data coordinates for loading or storing are (0,0), (1,0), (2,0), (3,0), (64,0), (65,0), (66,0), and (67,0). Another burst mode is the uint32 burst4 mode, which indicates that the data type loaded or stored is limited to uint32, and the number of data loaded or stored in one burst is 4. According to the coordinate mapping rules of this burst mode, the corresponding data coordinates for loading or storing are (0,0), (1,0), (32,0), and (33,0). The other burst modes are similar and will not be described in detail.

[0028] See Figure 3 , Figure 3 This is a flowchart of a weight matrix processing method provided in an embodiment of this application. The weight matrix processing method includes: S1. Load the target weight matrix; wherein, the target weight matrix is: a weight matrix of third precision packaged from a weight matrix of first precision into a weight matrix of third precision arranged in a preset format; S2. Unpack and dequantize the target weight matrix to obtain a weight matrix with second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; S3. Store the weight matrix of the second precision to the on-chip storage unit for loading into the tensor core to perform matrix multiplication operations.

[0029] It is worth noting that the embodiments of this application can be specifically applied to the computing unit (CU) of an artificial intelligence chip. By loading the target weight matrix, unpacking and dequantizing it, a second-precision weight matrix is ​​obtained. This second-precision weight matrix is ​​then stored in the on-chip storage unit. This allows the tcore within the computing unit to perform matrix multiplication operations when needed, by loading the second-precision weight matrix from the on-chip storage unit and multiplying it with the activation matrix loaded from the off-chip storage unit. Furthermore, the embodiments of this application pre-arrange the target weight matrix according to a preset format, ensuring the smooth execution of subsequent unpacking and dequantization.

[0030] This application does not specifically limit the first precision, second precision, and third precision. Generally speaking, the second precision can be any precision supported by tcore, such as int8 or BF16, so that the weight matrix of the second precision is compatible with the operation logic of tcore, thereby successfully completing matrix multiplication operations in tcore.

[0031] In one optional embodiment, the preset format includes: Each data point in the third precision weight matrix is ​​arranged according to the coordinate mapping rules of the second burst mode; wherein, the second burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst. Each data point in the third precision weight matrix is ​​arranged in the height dimension according to the coordinate mapping rules of the first burst mode; wherein, the first burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst.

[0032] In this embodiment, the data in the third precision weight matrix is ​​arranged according to the coordinate mapping rules of the second burst mode. The third precision data is arranged into multiple rows of data in the height dimension according to the coordinate mapping rules of the first burst mode to form the third precision weight matrix. This enables the loading or storage of data to be performed according to the corresponding burst mode, thereby improving the efficiency of data loading or storage.

[0033] Specifically, based on the coordinate mapping rules of the burst mode, the coordinates corresponding to the first burst mode and the second burst mode are found, and the weight matrix of the first precision is packed into a weight matrix of the third precision arranged in a preset format according to the coordinates.

[0034] Furthermore, the height dimension can also be arranged in conjunction with the number of execution units (eu) within the computing unit, so that one eu can load the data of the first burst mode once.

[0035] In one alternative embodiment, the first burst mode is specifically used to limit the data type loaded and the amount of data loaded in a single burst. The loaded target weight matrix includes: The target weight matrix is ​​loaded into the register according to the first burst mode.

[0036] This application embodiment utilizes the data type and data quantity defined by the first burst mode to load data, thereby improving the efficiency of data loading. Specifically, the first burst mode defines the data type to be loaded as having a first precision, in order to adapt to the precision of the target weight matrix.

[0037] In one optional embodiment, the second burst mode is used to limit the type of data stored and the amount of data stored in a single burst; The step of storing the weight matrix of the second precision to the on-chip storage unit includes: The weight matrix of the second precision is stored in the on-chip storage unit according to the second burst mode.

[0038] This application embodiment utilizes the data type and data quantity defined by the second burst mode to store data, thereby improving the efficiency of data storage. Specifically, the second burst mode defines the stored data type as second precision, so that it can be dequantized into data of second precision.

[0039] In one alternative embodiment, the target weight matrix is ​​pre-constructed on the host side and stored in an off-chip storage unit.

[0040] In this embodiment, the host computer pre-packages the first-precision weight matrix into a third-precision weight matrix arranged in a preset format according to a preset format, thereby forming a target weight matrix, which is stored in an off-chip storage unit so that the artificial intelligence chip can correctly unpack and dequantize it, ensuring the smooth execution of matrix multiplication operations.

[0041] Furthermore, when the host side packages the first-precision weight matrix, it fully considers the loading mode of the AI ​​chip's computing unit loading the third-precision data from the off-chip storage unit through the first burst mode, and the storage mode of the AI ​​chip's computing unit unpacking the third-precision data and dequantizing it into second-precision data and storing it according to the second burst mode. This forms a target weight matrix in which each data in the third-precision weight matrix is ​​arranged according to the coordinate mapping rules of the second burst mode, and each data in the third-precision weight matrix is ​​arranged in the height dimension according to the coordinate mapping rules of the first burst mode.

[0042] like Figure 4This application provides a weight matrix processing system comprising: a host computer, an off-chip storage unit, and an artificial intelligence chip. The host computer is used to construct the target weight matrix and store it in the off-chip storage unit.

[0043] Artificial intelligence chips consist of multiple streaming processor clusters (SPCs). One streaming processor cluster can handle one computational task, or multiple streaming processor clusters can handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory (e.g., high-bandwidth memory (HBM)).

[0044] A stream processor consists of multiple computing units, each of which performs arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0045] A computation unit includes multiple execution units and tensor cores. Execution units perform general-purpose computational tasks, while tensor cores perform specialized computational tasks. Additionally, a computation unit includes registers (e.g., ...). Figure 4 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the execution unit and the tensor core of that computing unit.

[0046] Each streaming processor cluster also provides a cache for caching data from the individual computing units within the streaming processor cluster.

[0047] In this embodiment of the application, the target weight matrix can be loaded into a register by the execution unit within the computing unit for unpacking, dequantization, and storage to an on-chip storage unit (e.g., ...). Figure 4 (Shared memory or cache in the memory); load the activation matrix of the off-chip storage unit and the weight matrix of the on-chip storage unit through the tensor core in the computing unit, and perform mma calculation.

[0048] In an optional embodiment, the step of unpacking and dequantizing the target weight matrix to obtain a weight matrix of second precision includes: The target weight matrix is ​​unpacked to obtain the weight matrix with the first precision. The weight matrix of the first precision is dequantized to obtain the weight matrix of the second precision and loaded into the register.

[0049] In this embodiment, the target weight matrix is ​​unpacked to obtain a weight matrix of first precision, which provides data support for subsequent dequantization. Then, the weight matrix of first precision is dequantized to obtain a weight matrix of second precision and loaded into a register, so that the weight matrix of second precision is subsequently stored from the register into the on-chip storage unit for loading by tcore.

[0050] In one alternative embodiment, the first precision is a 4-bit signed integer, the second precision is a 16-bit floating-point number, and the third precision is a 32-bit single-precision floating-point number.

[0051] In this embodiment, based on the coordinate mapping rules of the burst mode, the first burst mode is selected as uint32burst4 mode, and the second burst mode is BF16 burst8 mode. This embodiment processes the 32-unit weight data into BF16 weight data to be compatible with the Tcore's computational logic, thus enabling successful matrix multiplication in Tcore. Furthermore, loading or storing data using burst mode improves loading or storing efficiency.

[0052] In one optional embodiment, the first precision is a 4-bit signed integer, the second precision is an 8-bit signed integer, and the third precision is a 32-bit single-precision floating-point number.

[0053] In this embodiment, based on the coordinate mapping rules of burst modes, the first burst mode is selected as uint32burst4 mode, and the second burst mode is int8 burst8 mode. The specific process is as follows: On the host side, prepare the corresponding target weight matrix according to the arrangement requirements of loading uint32 burst4 mode and storing int8 burst8 mode. In the EU, load uint32 data into registers in burst4 format, unpack and dequantize to obtain burst8 formatted int8 data, store it in registers, and then store the data from the registers into on-chip memory; specifically, as follows... Figure 5 The system packages eight non-contiguous int4 data points into a single uint32 data point in parallel read order. An eu can load four uint32 data points. Subsequently, a single uint32 data point can be unpacked into eight int4 data points and dequantized into eight int8 data points, stored in on-chip memory. At this point, tcore loads the activation matrix (e.g., in BF16 format) from off-chip memory, loads the weight matrix in int8 format from on-chip memory, executes the mma calculation instructions, and outputs the result to the output matrix.

[0054] To support loading data using `int32 burst4` on hardware and arranging it in `int8burst8` mode after unpacking, the first burst mode is set to `uint32 burst4` mode, and the second burst mode is set to `int8burst8` mode. The resulting target matrix data is arranged as follows: Figure 6 As shown: A raw matrix is ​​composed of 512 int4 data points, with dimensions H (height dimension) of 512 and W (width dimension) of 1. Each box within the red area represents one data point (int4 precision), and the values ​​within represent the order and coordinates of the data within the raw matrix. Within each data point: a row of 8 int4 data points is combined in int8 burst8 format to form a uint32 data point. The height dimension is arranged in uint32 burst4 format, forming multiple rows of data. Specifically, this arrangement is based on the number of uint32 values ​​multiplied by the uint32 burst4 value, ensuring that when loading uint32 data and subsequently splitting burst4 data into rows for processing, each uint32 value receives corresponding continuous data in SIMD (Single Instruction, Multiple Data Parallel) format. For example, the first data point of uint0's burst4 is continuous with the first data points of uint1, uint2, ..., uint15. Only in this way can subsequent unpacking into uint32 data and dequantization processing be performed correctly.

[0055] In other words, the original matrix is ​​a matrix with H dimension of 512 and W dimension of 1 (precision int4). The values ​​in the cells within the red boxes represent their coordinates, ranging from 0 to 511. Taking the first row as an example, the coordinates of the eight data points 0, 1, 2, 3, 64, 65, 66, and 67 conform to the coordinate requirements of int8 burst8, and they will be packaged into a uint32 data. Such a uint32 data actually corresponds to... Figure 6 In one line of data, 512 int4s correspond to a total of 64 uint32s. These 64 uint32s are arranged in the height dimension in the form of uint32 burst4, which forms 64 rows of uint32s. Figure 6 In the `eu0` class, `d0`, `d1`, `d2`, and `d3` represent the `burst4` method of the `uint32` class, located at indexes 0, 1, 32, and 33 respectively, conforming to the `burst4` requirement of `uint32`. Each `eu` can load 4 rows through `burst4`, and with 16 `eu`s executing in parallel, a total of 64 rows of data can be loaded simultaneously.

[0056] Furthermore, for the original matrix that is 512 in dimension H and Y in dimension W, it is packaged as Y original matrices with 512 in dimension H and 1 in dimension W, such as... Figure 7As shown, different original matrices with H dimension of 512 and W dimension of 1 are distinguished by color and packaged. The total original matrix is ​​divided into 1, 2... original matrices with H dimension of 512 and W dimension of 1. For each original matrix with H dimension of 512 and W dimension of 1, it is arranged according to the above method. For example, the matrices in the yellow part of the original matrix are packaged to form the matrices in the yellow part of the target weight matrix. The specific arrangement of the matrices in the yellow part of the target weight matrix is ​​as follows. Figure 6 The green portion of the original matrix is ​​packaged to form the green portion of the target weight matrix. The specific arrangement of the green portion of the target weight matrix is ​​also as follows. Figure 6 .

[0057] Compared to the BF16 second precision scheme, the embodiments of this application reduce the data size of the weight matrix by half during multiple internal memory accesses, improving memory access efficiency. Furthermore, the data size of the weight matrix is ​​also reduced by half when performing mma calculations within the tcore, thereby improving tcore computational efficiency. Overall, memory access and computational efficiency are improved, and the overall operator performance is also enhanced. Actual measurements show that the int8 second precision scheme achieves a 2.33 to 2.41 times performance improvement compared to the BF16 second precision scheme for small shapes.

[0058] This application's embodiments fully leverage the computing power of Tcore mixed precision, significantly reducing the amount of computation and memory access data within the computing unit. The code is highly versatile and can be used to accelerate the MMA operator in any scenario that meets precision requirements. Furthermore, there is no additional data preparation or unpacking overhead from the host to the computing unit.

[0059] See Figure 8 , Figure 8 This is a structural block diagram of a weight matrix processing device 10 provided in an embodiment of this application. The weight matrix processing device 10 includes: A loading module is used to load a target weight matrix; wherein the target weight matrix is ​​a third-precision weight matrix packaged from a first-precision weight matrix and arranged in a preset format; The processing module is used to unpack and dequantize the target weight matrix to obtain a weight matrix with a second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; The save module is used to save the weight matrix of the second precision to the on-chip storage unit for loading into the tensor core to perform matrix multiplication operations.

[0060] Optionally, the preset format includes: Each data point in the third precision weight matrix is ​​arranged according to the coordinate mapping rules of the second burst mode; wherein, the second burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst. Each data point in the third precision weight matrix is ​​arranged in the height dimension according to the coordinate mapping rules of the first burst mode; wherein, the first burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst.

[0061] Optionally, the first burst mode is specifically used to limit the data types loaded and the amount of data loaded in a single burst; The loaded target weight matrix includes: The target weight matrix is ​​loaded into the register according to the first burst mode.

[0062] Optionally, the second burst mode is used to limit the data type stored and the amount of data stored in a single burst; The step of storing the weight matrix of the second precision to the on-chip storage unit includes: The weight matrix of the second precision is stored in the on-chip storage unit according to the second burst mode.

[0063] Optionally, the target weight matrix is ​​pre-constructed on the host side and stored in an off-chip storage unit: The weight matrices of the first precision are packaged to form the weight matrix of the third precision; The weight matrix of the third precision is arranged according to the preset format to form the target weight matrix.

[0064] Optionally, the step of unpacking and dequantizing the target weight matrix to obtain a weight matrix of second precision includes: The target weight matrix is ​​unpacked to obtain the weight matrix with the first precision. The weight matrix of the first precision is dequantized to obtain the weight matrix of the second precision and loaded into the register.

[0065] Optionally, the first precision is a 4-bit signed integer, the second precision is a 16-bit floating-point number, and the third precision is a 32-bit single-precision floating-point number.

[0066] Optionally, the first precision is a 4-bit signed integer, the second precision is an 8-bit signed integer, and the third precision is a 32-bit single-precision floating-point number.

[0067] It is worth noting that the working process of each module in the weight matrix processing device 10 described in this application embodiment can refer to the working process of the weight matrix processing method described in the above embodiment, and will not be repeated here.

[0068] This application provides a weight matrix processing device 10, which loads a target weight matrix. The target weight matrix is ​​a third-precision weight matrix packaged from a first-precision weight matrix and arranged in a preset format. The target weight matrix is ​​then unpacked and dequantized to obtain a second-precision weight matrix. The third precision is higher than the second precision, and the second precision is higher than the first precision. The second-precision weight matrix is ​​stored in an on-chip storage unit for loading into the tensor core for matrix multiplication. Therefore, this application embodiment processes the first-precision weight matrix from packaging to unpacking and finally to dequantization, ensuring that the dequantized second-precision weight matrix is ​​compatible with the Tcore's operational logic, thus enabling successful matrix multiplication within Tcore. Furthermore, the target matrix's preset format ensures smooth execution of subsequent unpacking and dequantization.

[0069] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the weight matrix processing method as described in any of the above embodiments.

[0070] Furthermore, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the weight matrix processing method as described in any of the above embodiments.

[0071] See Figure 9 , Figure 9 This is a structural block diagram of a weight matrix processing device 20 provided in an embodiment of this application. The weight matrix processing device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described weight matrix processing method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0072] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the weight matrix processing device 20.

[0073] The weight matrix processing device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the weight matrix processing device 20 and does not constitute a limitation on the weight matrix processing device 20. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the weight matrix processing device 20 may also include input / output devices, network access devices, buses, etc.

[0074] The processor 21 can be any one of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural Network Processing Unit), DPU (Deep Learning Processing Unit), APU (Accelerated Processing Unit), and GPGPU (General-Purpose Computing on Graphics Processing Unit). The processor 21 is the control center of the weight matrix processing device 20, connecting all parts of the weight matrix processing device 20 via various interfaces and lines.

[0075] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the weight matrix processing device 20 by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as matrix loading function, matrix unpacking function, etc.), etc.; the data storage area may store relevant data (such as intermediate results of matrix operations, matrix operation results, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0076] If the modules / units integrated in the weight matrix processing device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0077] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0078] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for processing a weight matrix, characterized in that, include: Load the target weight matrix; wherein, the target weight matrix is: a third-precision weight matrix packaged from a first-precision weight matrix and arranged in a preset format; The target weight matrix is ​​unpacked and dequantized to obtain a weight matrix with a second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; The weight matrix of the second precision is stored in the on-chip storage unit for loading into the tensor core to perform matrix multiplication operations.

2. The weight matrix processing method as described in claim 1, characterized in that, The preset format includes: Each data point in the third precision weight matrix is ​​arranged according to the coordinate mapping rules of the second burst mode; wherein, the second burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst. Each data point in the third precision weight matrix is ​​arranged in the height dimension according to the coordinate mapping rules of the first burst mode; wherein, the first burst mode is used to limit the data type loaded or stored, and the number of data loaded or stored in one burst.

3. The weight matrix processing method as described in claim 2, characterized in that, The first burst mode is specifically used to limit the data types loaded and the amount of data loaded in a single burst. The loaded target weight matrix includes: The target weight matrix is ​​loaded into the register according to the first burst mode.

4. The weight matrix processing method as described in claim 2, characterized in that, The second burst mode is used to limit the data types stored and the amount of data stored in a single burst; The step of storing the weight matrix of the second precision to the on-chip storage unit includes: The weight matrix of the second precision is stored in the on-chip storage unit according to the second burst mode.

5. The weight matrix processing method according to any one of claims 1-4, characterized in that, The target weight matrix is ​​pre-constructed on the host side and stored in an off-chip storage unit.

6. The weight matrix processing method as described in claim 1, characterized in that, The step of unpacking and inverse quantizing the target weight matrix to obtain a second-precision weight matrix includes: The target weight matrix is ​​unpacked to obtain the weight matrix with the first precision. The weight matrix of the first precision is dequantized to obtain the weight matrix of the second precision and loaded into the register.

7. The weight matrix processing method as described in claim 1, characterized in that, The first precision is a 4-bit signed integer, the second precision is a 16-bit floating-point number, and the third precision is a 32-bit single-precision floating-point number.

8. The weight matrix processing method as described in claim 1, characterized in that, The first precision is a 4-bit signed integer, the second precision is an 8-bit signed integer, and the third precision is a 32-bit single-precision floating-point number.

9. A weight matrix processing device, characterized in that, include: A loading module is used to load a target weight matrix; wherein the target weight matrix is ​​a third-precision weight matrix packaged from a first-precision weight matrix and arranged in a preset format; The processing module is used to unpack and dequantize the target weight matrix to obtain a weight matrix with a second precision; wherein the third precision is higher than the second precision, and the second precision is higher than the first precision; The save module is used to save the weight matrix of the second precision to the on-chip storage unit for loading into the tensor core to perform matrix multiplication operations.

10. A weight matrix processing device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the weight matrix processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the weight matrix processing method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the weight matrix processing method as described in any one of claims 1 to 8.

Images