A weighted inverse quantization matrix multiplication module and related equipment

Abstract

This invention proposes a weighted dequantization matrix multiplication module and related devices. A general-purpose matrix multiplication unit sends read control information to a direct memory access unit (DMI). The DMI generates a corresponding read request based on the read control information, sends the read request to off-chip memory, and receives target data from the off-chip memory. The target data includes at least target weight data, which is the weight data in the quantized weight matrix that matches both the target row identifier and the target column identifier. The DMI performs dequantization on the target weight data and provides the dequantized weight data to the general-purpose matrix multiplication unit. By having the DMI perform the dequantization and provide the dequantized weight data to the general-purpose matrix multiplication unit, the process does not require a global cache, thus avoiding the use of global cache memory bandwidth and preventing additional and unnecessary power consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of chips, in particular to a weight dequantization matrix multiplication module and related equipment. BACKGROUND

[0002] In recent years, with the development of large models, the model volume is getting larger and larger, so the model volume is often reduced by quantizing the trained high-precision weights into a low-precision format, thereby reducing the bandwidth occupation and energy consumption of reading weights. When inferring, the quantized weights are read in, and then weight dequantization is performed to maintain the output precision of the neural network.

[0003] Weight dequantization is required during inference, and the weight dequantization is usually completed by a vector processing unit. Since the vector processing unit and the general matrix multiplication unit (GEMM) are usually located at different positions in the hardware of the neural network processor (NPU), the data interaction between them needs to be realized through a global cache, which leads to increased memory access and additional energy consumption of moving data, which has become a problem that technicians in the field are concerned about. SUMMARY

[0004] The purpose of the present application is to provide a weight dequantization matrix multiplication module and related equipment to improve the above problems.

[0005] In order to achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a weight dequantization matrix multiplication module, comprising a direct memory access unit and a general matrix multiplication unit.

[0007] The general matrix multiplication unit is configured to send read control information to the direct memory access unit, and the read control information comprises a target row identifier, a target column identifier, and a dequantization mode.

[0008] The direct memory access unit is configured to generate a corresponding read request according to the read control information, send the read request to an off-chip memory, and receive target data fed back by the off-chip memory, wherein the target data at least comprises target weight data, and the target weight data is weight data in a quantized weight matrix that is matched with both the target row identifier and the target column identifier.

[0009] The direct memory access unit is configured to perform dequantization processing on the target weight data, and provide the dequantized weight data to the general matrix multiplication unit.

[0010] Optionally, the direct memory access unit comprises a control device, a scaling factor cache, a zero point cache, and N weight dequantization devices, and N≥2.

[0011] The control device is connected with the scaling factor buffer, the zero point buffer and N weight dequantization devices respectively, and each weight dequantization device is connected with the scaling factor buffer and the zero point buffer.

[0012] Optionally, when the read control information is the layer-by-layer dequantization mode and the asymmetric dequantization mode, the read control information further includes scaling factors corresponding to the quantized weight matrix when performing dequantization.

[0013] The control device is configured to write the scaling factors in the read control information into the scaling factor buffer and write the zero points in the read control information into the zero point buffer.

[0014] The control device is further configured to generate a weight read request according to the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0015] The control device is configured to write the obtained target weight data into the corresponding weight dequantization device.

[0016] The weight dequantization device is configured to perform dequantization processing on the obtained target weight data according to the scaling factors in the scaling factor buffer and the zero points in the zero point buffer, and provide the dequantized weight data to the general matrix multiplication unit.

[0017] Optionally, when the read control information is the layer-by-layer dequantization mode and the symmetric dequantization mode, the read control information further includes scaling factors corresponding to the quantized weight matrix when performing dequantization.

[0018] The control device is configured to write the scaling factors in the read control information into the scaling factor buffer.

[0019] The control device is further configured to generate a weight read request according to the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0020] The control device is configured to write the obtained target weight data into the corresponding weight dequantization device.

[0021] The weight dequantization device is configured to perform dequantization processing on the obtained target weight data according to the scaling factors in the scaling factor buffer, and provide the dequantized weight data to the general matrix multiplication unit.

[0022] Optionally, when the reading control information is in channel-by-channel dequantization mode and asymmetric dequantization mode, the target data further includes first-type reference factor data, which includes the scaling factor and zero point corresponding to the target column in the weight matrix when performing dequantization.

[0023] The control device is used to generate a first type of parameter read request including the target column identifier, send the first type of parameter read request to the off-chip memory, and receive the first type of reference factor data fed back by the off-chip memory;

[0024] The control device is used to write the scaling factor in the first type of reference factor data into the scaling factor cache, and to write the zero point in the first type of reference factor data into the zero point cache.

[0025] The control device is also configured to generate a weighted read request including the target row identifier and the target column identifier, send the weighted read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory;

[0026] The control device is used to write the acquired target weight data into the corresponding weight dequantization device;

[0027] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache and the zero point in the zero point cache, and provide the dequantized weight data to the general matrix multiplication unit.

[0028] Optionally, when the reading control information is in channel-by-channel dequantization mode and symmetric dequantization mode, the target data further includes second-type reference factor data, which includes the scaling factor corresponding to the target column in the weight matrix when performing dequantization.

[0029] The control device is used to generate a second type of parameter read request including the target column identifier, send the second type of parameter read request to the off-chip memory, and receive the second type of reference factor data fed back by the off-chip memory;

[0030] The control device is used to write the scaling factor in the second type of reference factor data into the scaling factor cache;

[0031] The control device is also configured to generate a weighted read request including the target row identifier and the target column identifier, send the weighted read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory;

[0032] The control device is used to write the acquired target weight data into the corresponding weight dequantization device;

[0033] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache, and provide the dequantized weight data to the general matrix multiplication unit.

[0034] Optionally, when the read control information is in block dequantization mode and asymmetric dequantization mode, the target data further includes third-type reference factor data, which includes the scaling factor and zero point corresponding to the target column in the weight matrix when performing dequantization;

[0035] The control device is used to generate a third type of parameter read request including the target column identifier and the target row identifier, send the third type of parameter read request to the off-chip memory, and receive the third type of reference factor data fed back by the off-chip memory;

[0036] The control device is used to write the scaling factor in the third type of reference factor data into the scaling factor cache, and to write the zero point in the third type of reference factor data into the zero point cache;

[0037] The control device is also configured to generate a weighted read request including the target row identifier and the target column identifier, send the weighted read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory;

[0038] The control device is used to write the acquired target weight data into the corresponding weight dequantization device;

[0039] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache and the zero point in the zero point cache, and provide the dequantized weight data to the general matrix multiplication unit.

[0040] Optionally, when the read control information is in block dequantization mode and symmetric dequantization mode, the target data further includes fourth type reference factor data, which includes the scaling factor corresponding to the target column in the weight matrix when performing dequantization.

[0041] The control device is used to generate a fourth type of parameter read request including the target column identifier and the target row identifier, send the fourth type of parameter read request to the off-chip memory, and receive the fourth type of reference factor data fed back by the off-chip memory;

[0042] The control device is used to write the scaling factor in the fourth type of reference factor data into the scaling factor cache;

[0043] The control device is also configured to generate a weighted read request including the target row identifier and the target column identifier, send the weighted read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory;

[0044] The control device is used to write the acquired target weight data into the corresponding weight dequantization device;

[0045] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache, and provide the dequantized weight data to the general matrix multiplication unit.

[0046] Optionally, the weight dequantization device includes a selector, a subtractor, a converter, and a multiplier;

[0047] The input terminal of the selector serves as the input terminal of the weight dequantization device. The first output terminal of the selector is connected to the first input terminal of the subtractor. The second input terminal of the subtractor is connected to the zero-point buffer. The second output terminal of the selector and the output terminal of the subtractor are connected to the input terminal of the converter. The output terminal of the converter is connected to the first input terminal of the multiplier. The second input terminal of the multiplier is connected to the scaling factor buffer. The output terminal of the multiplier serves as the output terminal of the weight dequantization device. The control terminal of the selector is connected to the control device.

[0048] The control device is used to control the input terminal of the selector to be connected to the first output terminal when the read control information is in asymmetric inverse quantization mode, and to control the input terminal of the selector to be connected to the second output terminal when the read control information is in symmetric inverse quantization mode.

[0049] The subtractor is used to perform zero-point correction when the target weight data is obtained, and to transmit the corrected weight data to the converter.

[0050] The converter is used to perform format conversion processing on the received weight data and transmit the converted weight data to the multiplier.

[0051] The multiplier is used to scale the acquired weight data in conjunction with a scaling factor to obtain the dequantized weight data.

[0052] Secondly, embodiments of the present invention provide a neural network processor, which includes the aforementioned weight dequantization matrix multiplication module.

[0053] Thirdly, embodiments of the present invention provide an electronic device including the aforementioned neural network processor.

[0054] Compared to existing technologies, the weighted dequantization matrix multiplication module and related devices provided in this invention include a general matrix multiplication unit that sends read control information to a direct memory access unit. The read control information includes a target row identifier, a target column identifier, and a dequantization mode. The direct memory access unit generates a corresponding read request based on the read control information, sends the read request to an external memory, and receives target data from the external memory. The target data includes at least target weight data, which is the weight data in the quantized weight matrix that matches both the target row identifier and the target column identifier. The direct memory access unit performs dequantization on the target weight data and provides the dequantized weight data to the general matrix multiplication unit, which then performs the matrix multiplication operation. Since the dequantization is performed by the direct memory access unit, and the dequantized weight data is provided to the general matrix multiplication unit, it does not require a global cache, thus avoiding the use of global cache memory bandwidth and preventing additional and unnecessary power consumption.

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of the connection of the weight dequantization matrix multiplication module provided in an embodiment of the present invention.

[0058] Figure 2 This is a connection diagram of the direct memory access unit provided in an embodiment of the present invention.

[0059] Figure 3 This is one of the structural schematic diagrams of the weighted dequantization device provided in an embodiment of the present invention.

[0060] Figure 4 This is a second schematic diagram of the structure of the weighted dequantization device provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0063] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0064] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0066] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0067] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0068] Please refer to Figure 1 , Figure 1 This is a connection diagram of the weight dequantization matrix multiplication module provided in an embodiment of the present invention. The weight dequantization matrix multiplication module includes a direct memory access unit (also known as DMA) and a general matrix multiplication unit (also known as GEMM). The direct memory access unit is connected to the general matrix multiplication unit, and the direct memory access unit is also connected to an off-chip memory, which may be, but is not limited to, dynamic random access memory (DDR).

[0069] The general matrix multiplication unit is used to send read control information to the direct memory access unit. The read control information includes the target row identifier, the target column identifier, and the dequantization mode.

[0070] The target row identifier (general matrix multiplication unit) is the identifier of the row of the weight data to be called in the quantized weight matrix, and the target column identifier (general matrix multiplication unit) is the identifier of the column of the weight data to be called in the quantized weight matrix.

[0071] The direct memory access unit is used to generate corresponding read requests based on read control information, send the read requests to off-chip memory, and receive target data fed back from off-chip memory.

[0072] The target data includes at least target weight data, which is the weight data in the quantized weight matrix that matches both the target row identifier and the target column identifier. The read request includes any one or more of the following: the first type of parameter read request, the second type of parameter read request, the third type of parameter read request, the fourth type of parameter read request, and the weight read request.

[0073] The direct memory access unit is used to dequantize the target weight data and provide the dequantized weight data to the general matrix multiplication unit, which then performs the matrix multiplication operation.

[0074] In the weight dequantization matrix multiplication module provided in this embodiment of the invention, the dequantization process is completed by the direct memory access unit, and the dequantized weight data is provided to the general matrix multiplication unit. It does not need to go through the global cache, does not occupy the memory access bandwidth of the global cache, and avoids generating additional and unnecessary power consumption.

[0075] In the weighted inverse quantization matrix multiplication module provided in this embodiment of the invention, the PE (processing unit) in the general matrix multiplication unit (also known as GEMM) supports integer multiplication-addition and floating-point multiplication-addition operations.

[0076] Building upon the foregoing, this embodiment of the invention also provides an optional implementation method for the direct memory access unit, please refer to... Figure 2 , Figure 2 This is a connection diagram of the direct memory access unit provided in an embodiment of the present invention.

[0077] The direct memory access unit includes a control unit, a scaling factor cache, a zero-point cache, and N weighted dequantization units, where N ≥ 2.

[0078] The control device is connected to the scaling factor cache, the zero-point cache, and N weight dequantization devices, respectively. Each weight dequantization device is connected to the scaling factor cache and the zero-point cache.

[0079] N weight dequantization devices can achieve parallel dequantization processing, and the parallelism gradually increases with the value of N. The number of weight dequantization devices in the Direct Memory Access (DMA) unit matches the rate at which the General Matrix Multiplication (GEMM) unit consumes weights, and the parallelism is much greater than that of traditional vector processing units.

[0080] Based on the foregoing, regarding the working process of the direct memory access unit, this embodiment of the invention also provides an optional implementation method, please refer to the following text.

[0081] When the control information is read in both layer-by-layer dequantization mode and asymmetric dequantization mode (both conditions are met), the control information also includes the scaling factor and zero point of the quantized weight matrix when performing dequantization.

[0082] When the data format of the quantized weight matrix is ​​UINT4 (unsigned 4-bit integer) or UINT8 (unsigned 8-bit integer), the dequantization mode in the read control information is asymmetric dequantization mode.

[0083] The control device is used to write the scaling factor in the read control information into the scaling factor cache, and to write the zero point in the read control information into the zero point cache.

[0084] After completing the scaling factor cache and zero-point cache configuration, the control unit is also used to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0085] The control device is used to write the acquired target weight data into the corresponding weight dequantization device.

[0086] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache and the zero point in the zero point cache to obtain the dequantized weight data, and then provide the dequantized weight data to the general matrix multiplication unit.

[0087] When the control information is read in both layer-by-layer dequantization and symmetric dequantization modes (both conditions are met), the control information also includes the scaling factor corresponding to the quantized weight matrix during dequantization.

[0088] When the data format of the quantized weight matrix is ​​INT4 (signed 4-bit integer), INT8 (signed 8-bit integer), FP4 (4-bit floating-point number), or FP8 (8-bit floating-point number), the dequantization mode in the read control information is the symmetric dequantization mode.

[0089] The control device is used to write the scaling factor from the read control information into the scaling factor cache.

[0090] After completing the scaling factor cache configuration, the control unit is also used to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0091] The control device is used to write the acquired target weight data into the corresponding weight dequantization device.

[0092] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache to obtain the dequantized weight data, and then provide the dequantized weight data to the general matrix multiplication unit.

[0093] When reading control information in channel-wise dequantization mode and asymmetric dequantization mode (simultaneously satisfying channel-wise dequantization mode and asymmetric dequantization mode), the target data also includes first-type reference factor data, which includes the scaling factor and zero point corresponding to the target column in the weight matrix when performing dequantization.

[0094] The control device is used to generate a first type of parameter read request including the target column identifier, send the first type of parameter read request to the off-chip memory, and receive the first type of reference factor data fed back by the off-chip memory.

[0095] The control device is used to write the scaling factor in the first type of reference factor data into the scaling factor cache and to write the zero point in the first type of reference factor data into the zero point cache.

[0096] After completing the scaling factor cache and zero-point cache configuration, the control unit is also used to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0097] The control device is used to write the acquired target weight data into the corresponding weight dequantization device.

[0098] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache and the zero point in the zero point cache to obtain the dequantized weight data, and then provide the dequantized weight data to the general matrix multiplication unit.

[0099] When reading control information in both channel-based dequantization and symmetric dequantization modes (simultaneously satisfying both channel-based and symmetric dequantization modes), the target data also includes second-type reference factor data, which includes the scaling factor corresponding to the target column in the weight matrix when performing dequantization.

[0100] The control device is used to generate a second type of parameter read request including the target column identifier, send the second type of parameter read request to the off-chip memory, and receive the second type of reference factor data fed back by the off-chip memory.

[0101] The control device is used to write the scaling factor in the second type of reference factor data into the scaling factor cache.

[0102] After completing the scaling factor cache configuration, the control unit is also used to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0103] The control device is used to write the acquired target weight data into the corresponding weight dequantization device.

[0104] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache to obtain the dequantized weight data, and then provide the dequantized weight data to the general matrix multiplication unit.

[0105] When reading control information in block dequantization mode and asymmetric dequantization mode (satisfying both block dequantization mode and asymmetric dequantization mode), the target data also includes third-class reference factor data, which includes the scaling factor and zero point corresponding to the target column in the weight matrix when performing dequantization.

[0106] The control unit is used to generate a third type of parameter read request including a target column identifier and a target row identifier, send the third type of parameter read request to the off-chip memory, and receive the third type of reference factor data fed back from the off-chip memory.

[0107] The control device is used to write the scaling factor in the third type of reference factor data into the scaling factor cache and to write the zero point in the third type of reference factor data into the zero point cache.

[0108] After completing the scaling factor cache and zero-point cache configuration, the control unit is also used to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0109] The control device is used to write the acquired target weight data into the corresponding weight dequantization device.

[0110] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache and the zero point in the zero point cache to obtain the dequantized weight data, and then provide the dequantized weight data to the general matrix multiplication unit.

[0111] When reading control information in block dequantization mode and symmetric dequantization mode (satisfying both block dequantization mode and symmetric dequantization mode), the target data also includes fourth type reference factor data, which includes the scaling factor corresponding to the target column in the weight matrix when performing dequantization.

[0112] The control unit is used to generate a fourth type of parameter read request, including a target column identifier and a target row identifier, and send the fourth type of parameter read request to the off-chip memory, and receive the fourth type of reference factor data fed back from the off-chip memory.

[0113] The control device is used to write the scaling factor in the fourth type of reference factor data into the scaling factor cache.

[0114] After completing the scaling factor cache configuration, the control unit is also used to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory.

[0115] The control device is used to write the acquired target weight data into the corresponding weight dequantization device;

[0116] The weight dequantization device is used to dequantize the acquired target weight data according to the scaling factor in the scaling factor cache to obtain the dequantized weight data, and then provide the dequantized weight data to the general matrix multiplication unit.

[0117] Building upon the foregoing, this invention also provides an optional implementation method for the weighted inverse quantization device. Please refer to [link / reference]. Figure 3 , Figure 3 This is one of the structural schematic diagrams of the weighted dequantization device provided in an embodiment of the present invention.

[0118] The weighted dequantization device includes a selector, a subtractor, a converter, and a multiplier.

[0119] The input terminal of the selector serves as the input terminal of the weight dequantization device. The first output terminal of the selector is connected to the first input terminal of the subtractor. The second input terminal of the subtractor is connected to the zero-point buffer (for accessing the zero point). The second output terminal of the selector and the output terminal of the subtractor are connected to the input terminal of the converter. The output terminal of the converter is connected to the first input terminal of the multiplier. The second input terminal of the multiplier is connected to the scaling factor buffer (for accessing the scaling factor). The output terminal of the multiplier serves as the output terminal of the weight dequantization device. The control terminal of the selector is connected to the control device.

[0120] The control device is used to connect the input terminal of the control selector to the first output terminal when the read control information is in asymmetric inverse quantization mode (corresponding to the asymmetric inverse quantization path in the figure), and to connect the input terminal of the control selector to the second output terminal when the read control information is in symmetric inverse quantization mode (corresponding to the symmetric inverse quantization path in the figure).

[0121] The subtractor is used to perform zero-point correction when the target weight data is obtained, and then transmits the corrected weight data to the converter.

[0122] Zero-point correction refers to subtracting zero from the target weight data to obtain the corrected weight data.

[0123] The converter is used to perform format conversion processing on the received weight data (corrected weight data or target data transmitted by the selector) and transmit the format-converted weight data to the multiplier.

[0124] It should be understood that the converted weight data (which can be in FP16 (16-bit floating-point) format) can be adapted for floating-point multiplication and addition.

[0125] The multiplier is used to scale the acquired weight data in combination with a scaling factor to obtain the dequantized weight data.

[0126] Scaling refers to multiplying the weight data obtained by the multiplier with a scaling factor to obtain the dequantized weight data.

[0127] Building upon the foregoing, this invention also provides an optional implementation method for the weighted inverse quantization device. Please refer to [link / reference]. Figure 4 , Figure 4 This is a second schematic diagram of the structure of the weight dequantization device provided in an embodiment of the present invention. The weight dequantization device includes a selector, a subtractor, a converter, and a multiplier.

[0128] The first input of the selector and the first input of the subtractor serve as the inputs of the weight dequantization device (used to input the target weight data). The second input of the subtractor is connected to the zero-point buffer. The output of the subtractor is connected to the second input of the selector. The output of the selector is connected to the input of the converter. The output of the converter is connected to the first input of the multiplier. The second input of the multiplier is connected to the scaling factor buffer. The output of the multiplier serves as the output of the weight dequantization device. The control terminal of the selector is connected to the control device.

[0129] The control device is used to connect the second input terminal and the output terminal of the control selector when the read control information is in asymmetric inverse quantization mode, and to connect the first input terminal and the output terminal of the control selector when the read control information is in symmetric inverse quantization mode.

[0130] The subtractor is used to perform zero-point correction when the target weight data is obtained, and then transmits the corrected weight data to the converter.

[0131] The converter is used to perform format conversion on the received weight data and transmit the converted weight data to the multiplier.

[0132] The multiplier is used to scale the acquired weight data using a scaling factor to obtain the dequantized weight data. Since the weight dequantization device has already completed the format conversion, the multiplier in the general matrix multiplication unit can be adapted.

[0133] In this embodiment of the invention, the direct memory access unit can first read the scaling factor and zero point from the off-chip memory according to the read control information. Therefore, this implementation can support weight dequantization of different granularities. The weight dequantization device not only supports weight dequantization of various data types, but can also be bypassed. This allows GEMM to handle both pure integer or pure floating-point multiplication and addition for traditional small models, and also to achieve mixed-precision multiplication and addition required by large models with the help of the weight dequantization device, resulting in high flexibility and strong versatility.

[0134] In the weight dequantization matrix multiplication module provided in this invention, dequantization is performed while the weight data is being moved, and the dequantized weight data is directly sent to the general matrix multiplication unit (GEMM). This implementation supports dequantization of weights of types INT4, UINT4, INT8, UNIT8, FP4, and FP8. Dequantization of weights of different data types can share some hardware resources, and this implementation supports dequantization at multiple granularities, as well as symmetric and asymmetric quantization. It can also bypass the dequantization function, making it highly versatile and flexible.

[0135] This invention also provides a neural network processor, which includes the aforementioned weight dequantization matrix multiplication module.

[0136] This invention also provides an electronic device, which may be a system-on-a-chip, a mobile phone, a computer, or the above-mentioned neural network processor.

[0137] In summary, the weighted dequantization matrix multiplication module and related devices provided in this embodiment of the invention include a general matrix multiplication unit that sends read control information to a direct memory access unit. The read control information includes a target row identifier, a target column identifier, and a dequantization mode. The direct memory access unit generates a corresponding read request based on the read control information, sends the read request to an off-chip memory, and receives target data from the off-chip memory. The target data includes at least target weight data, which is the weight data in the quantized weight matrix that matches both the target row identifier and the target column identifier. The direct memory access unit performs dequantization processing on the target weight data and provides the dequantized weight data to the general matrix multiplication unit, which then performs the matrix multiplication operation. Since the dequantization processing is performed by the direct memory access unit and the dequantized weight data is provided to the general matrix multiplication unit, it does not require a global cache, thus avoiding the use of global cache memory access bandwidth and preventing additional and unnecessary power consumption.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0139] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A weight dequantization matrix multiplication module, comprising: The direct memory access unit and the general matrix multiplication unit are included; The general matrix multiplication unit is configured to send read control information to the direct memory access unit, the read control information including a target row identifier, a target column identifier, and a dequantization mode; The direct memory access unit is configured to generate a corresponding read request according to the read control information, send the read request to an off-chip memory, and receive target data fed back by the off-chip memory, the target data including at least target weight data, the target weight data being weight data in a quantized weight matrix that is simultaneously matched with the target row identifier and the target column identifier; The direct memory access unit is configured to perform dequantization processing on the target weight data and provide dequantized weight data to the general matrix multiplication unit; The direct memory access unit includes a control device, a scaling factor cache, a zero point cache, and N weight dequantization devices, N≥2; The control device is connected with the scaling factor cache, the zero point cache, and the N weight dequantization devices respectively, and each weight dequantization device is connected with the scaling factor cache and the zero point cache.

2. The weight dequantization matrix multiplication module of claim 1, wherein, When the read control information is a layer-based dequantization mode and an asymmetric dequantization mode, the read control information further includes a scaling factor and a zero point corresponding to the execution of dequantization on the quantized weight matrix; The control device is configured to write the scaling factor in the read control information into the scaling factor cache and write the zero point in the read control information into the zero point cache; The control device is further configured to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory; The control device is configured to write the obtained target weight data into a corresponding weight dequantization device; The weight dequantization device is configured to perform dequantization processing on the obtained target weight data according to the scaling factor in the scaling factor cache and the zero point in the zero point cache, and provide dequantized weight data to the general matrix multiplication unit.

3. The weight dequantization matrix multiplication module of claim 1, wherein, When the read control information is a channel-based dequantization mode and an asymmetric dequantization mode, the target data further includes first type reference factor data, the first type reference factor data including a scaling factor and a zero point corresponding to the execution of dequantization on a target column in a weight matrix; The control device is configured to generate a first type parameter read request including the target column identifier, send the first type parameter read request to the off-chip memory, and receive first type reference factor data fed back by the off-chip memory; The control device is configured to write the scaling factor in the first type reference factor data into the scaling factor cache and write the zero point in the first type reference factor data into the zero point cache; The control device is further configured to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory; The control device is configured to write the obtained target weight data to a corresponding weight dequantization device; The weight dequantization device is configured to perform dequantization processing on the obtained target weight data according to the scaling factors in the scaling factor cache and the zero points in the zero point cache, and provide the dequantized weight data to the general matrix multiplication unit.

4. The weight dequantization matrix multiplication module of claim 1, wherein, When the read control information is the channel-by-channel dequantization mode and the symmetric dequantization mode, the target data further includes second-type reference factor data, the second-type reference factor data including scaling factors corresponding to dequantization of target columns in the weight matrix; The control device is configured to generate a second-type parameter read request including the target column identifier, send the second-type parameter read request to the off-chip memory, and receive second-type reference factor data fed back by the off-chip memory; The control device is configured to write the scaling factors in the second-type reference factor data to the scaling factor cache; The control device is further configured to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory; The control device is configured to write the obtained target weight data to a corresponding weight dequantization device; The weight dequantization device is configured to perform dequantization processing on the obtained target weight data according to the scaling factors in the scaling factor cache and the zero points in the zero point cache, and provide the dequantized weight data to the general matrix multiplication unit.

5. The weight dequantization matrix multiplication module of claim 1, wherein, When the read control information is the block-by-block dequantization mode and the asymmetric dequantization mode, the target data further includes third-type reference factor data, the third-type reference factor data including scaling factors corresponding to dequantization of target columns in the weight matrix and zero points; The control device is configured to generate a third-type parameter read request including the target column identifier and the target row identifier, send the third-type parameter read request to the off-chip memory, and receive third-type reference factor data fed back by the off-chip memory; The control device is configured to write the scaling factors in the third-type reference factor data to the scaling factor cache, and write the zero points in the third-type reference factor data to the zero point cache; The control device is further configured to generate a weight read request including the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory; The control device is configured to write the obtained target weight data to a corresponding weight dequantization device; The weight dequantization device is configured to perform dequantization processing on the obtained target weight data according to the scaling factors in the scaling factor cache and the zero points in the zero point cache, and provide the dequantized weight data to the general matrix multiplication unit.

6. The weight dequantization matrix multiplication module of claim 1, wherein, When the read control information is the block inverse quantization mode and the symmetric inverse quantization mode, the target data further comprises fourth type reference factor data, the fourth type reference factor data comprising a scaling factor corresponding to inverse quantization of a target column in a weight matrix; The control device is configured to generate a fourth type parameter read request comprising the target column identifier and the target row identifier, and send the fourth type parameter read request to the off-chip memory, and receive the fourth type reference factor data fed back by the off-chip memory; The control device is configured to write the scaling factor in the fourth type reference factor data into the scaling factor cache; The control device is further configured to generate a weight read request comprising the target row identifier and the target column identifier, send the weight read request to the off-chip memory, and receive the target weight data fed back by the off-chip memory; The control device is configured to write the obtained target weight data into the corresponding weight inverse quantization device; The weight inverse quantization device is configured to perform inverse quantization processing on the obtained target weight data according to the scaling factor in the scaling factor cache, and provide the inverse quantized weight data to the general matrix multiplication unit.

7. The weight dequantization matrix multiplication module of claim 1, wherein, The weight inverse quantization device comprises a selector, a subtractor, a converter and a multiplier; An input terminal of the selector is an input terminal of the weight inverse quantization device, a first output terminal of the selector is connected to a first input terminal of the subtractor, a second input terminal of the subtractor is connected to the zero point cache, a second output terminal of the selector and an output terminal of the subtractor are connected to an input terminal of the converter, an output terminal of the converter is connected to a first input terminal of the multiplier, a second input terminal of the multiplier is connected to the scaling factor cache, an output terminal of the multiplier is an output terminal of the weight inverse quantization device, and a control terminal of the selector is connected to the control device; The control device is configured to control the input terminal and the first output terminal of the selector to be conductive when the read control information is the asymmetric inverse quantization mode, and control the input terminal and the second output terminal of the selector to be conductive when the read control information is the symmetric inverse quantization mode; The subtractor is configured to perform zero point correction on the obtained target weight data, and transmit the corrected weight data to the converter; The converter is configured to perform format conversion processing on the received weight data, and transmit the format converted weight data to the multiplier; The multiplier is configured to perform scaling processing on the obtained weight data in combination with the scaling factor, to obtain inverse quantized weight data.

8. A neural network processor, comprising: The neural network processor comprises the weight inverse quantization matrix multiplication module of any one of claims 1-7.

9. An electronic device, comprising: The neural network processor comprises the neural network processor of claim 8.

Images