Apparatus and method for dequantizing data

Abstract

The present disclosure provides an apparatus and method for dequantizing data, wherein the apparatus may be implemented in a combined processing apparatus, and the combined processing apparatus may further comprise a universal interconnection interface and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to jointly complete a computing operation specified by a user. The combined processing apparatus may further comprise a storage apparatus, wherein the storage apparatus is connected to a device and to the other processing apparatuses, and is configured for storing data of the device and the other processing apparatuses.

Description

Apparatus and method for dequantizing data Cross-reference to related applications

[0001] The present application claims priority to Chinese Patent Application No. 202410813872.3, filed on June 21, 2024, entitled “Apparatus and method for dequantizing data” and Chinese Patent Application No. 202411347345.4, filed on September 25, 2024, entitled “Apparatus and method for dequantizing data”. TECHNICAL FIELD

[0002] The present disclosure relates to the field of artificial intelligence, and more particularly, to the conversion of data types, and especially to an apparatus and method for dequantizing data. BACKGROUND

[0003] With the rapid development of deep learning technology, the size and complexity of models are increasing, which leads to a sharp increase in the demand for computing resources when deploying and running the models. As one of the key methods to solve this problem, quantization technology can effectively reduce the amount of computation and save computing resources by reducing the data precision of the operation data in the model.

[0004] However, existing quantization technologies have the problem of a sharp decline in task accuracy when processing original data into low-precision data types, especially ultra-low-precision data types. Therefore, a data quantization scheme is provided to ensure task accuracy while quantizing data.

[0005] In addition, when restoring low-bit-width data to high-bit-width data, i.e., when dequantizing data, a suitable scheme is also needed to make the restored data as close as possible to or reach the original precision. SUMMARY

[0006] One object of the present disclosure is to provide a scheme that can efficiently dequantize data.

[0007] According to a first aspect of the present disclosure, an apparatus for dequantizing target data is provided, comprising: a control circuit configured to parse a conversion instruction, the conversion instruction being used for data conversion and dequantizing the target data in the process of data conversion; a processing circuit configured to: perform a first conversion on the target data to obtain to-be-dequantized data; perform a second conversion on a scaling coefficient used for dequantization to obtain a dequantization scaling coefficient, wherein the second conversion corresponds to the first conversion; and perform a dequantization operation on the to-be-dequantized data through the dequantization scaling coefficient to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

[0008] According to a second aspect of this disclosure, an apparatus for inverse quantization of target data is provided, comprising: a first conversion component, a second conversion component, and an inverse quantization component, wherein the first conversion component is used to perform a first conversion on the data to obtain data to be inverse quantized; the second conversion component is used to perform a second conversion on a scaling factor used for inverse quantization to obtain an inverse quantization scaling factor, wherein the second conversion corresponds to the first conversion; and the inverse quantization component is connected to the first conversion component and the second conversion component respectively, and is used to perform an inverse quantization operation on the data to be inverse quantized using the inverse quantization scaling factor to obtain inverse quantized data, wherein the bit width of the inverse quantized data is greater than the bit width of the target data.

[0009] According to a third aspect of this disclosure, a method for dequantizing data is provided, comprising: performing a first transformation on the target data to obtain data to be dequantized; performing a second transformation on a scaling factor used for dequantization to obtain a dequantization scaling factor, wherein the second transformation corresponds to the first transformation; and performing a dequantization operation on the data to be dequantized using the dequantization scaling factor to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

[0010] According to a fourth aspect of this disclosure, an integrated circuit device is provided, including the device described above.

[0011] According to a fifth aspect of this disclosure, a board is provided, including the integrated circuit device described above.

[0012] According to a sixth aspect of this disclosure, a computing device for inverse quantization is provided, comprising: at least one processor; and at least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform the method as described above.

[0013] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided, wherein program instructions are stored, which, when loaded and executed by a processor, cause the processor to perform the method described above.

[0014] At least one beneficial effect of the technical solution disclosed herein is that it can improve the accuracy of data dequantization as much as possible and reduce the information loss caused by data quantization. Attached Figure Description

[0015] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0016] Figure 1 shows a schematic diagram of the structure of a board according to an embodiment of the present disclosure;

[0017] Figure 2 shows a schematic diagram of a combined processing apparatus according to one embodiment of the present disclosure;

[0018] Figure 3 shows a schematic diagram of the internal structure of a computing device according to one embodiment of the present disclosure;

[0019] Figure 4a shows a schematic diagram of the internal architecture of a processor core according to one embodiment of the present disclosure;

[0020] Figure 4b shows a schematic diagram of the MX quantization data format;

[0021] Figure 5 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure;

[0022] Figure 6 shows a schematic block diagram of the comparison circuit of some embodiments of this disclosure;

[0023] Figure 7 illustrates an apparatus for inverse quantization of target data according to one embodiment of the present disclosure;

[0024] Figure 8 shows a flowchart of determining the scaling factor according to one embodiment of the present disclosure;

[0025] Figures 9a and 9b illustrate an example of converting target data and scaling factors according to an embodiment of the present disclosure;

[0026] Figure 10 shows a schematic diagram of a pipelined parallel operation according to one embodiment of the present disclosure;

[0027] Figure 11 illustrates an apparatus for inverse quantization of target data according to one embodiment of the present disclosure;

[0028] Figure 12 shows a flowchart of a method for dequantizing data;

[0029] Figure 13 shows a block diagram of an apparatus for performing data conversion according to another embodiment of the present disclosure;

[0030] Figure 14 shows a schematic diagram of the structure of a data processing apparatus 5100 according to one embodiment of the present disclosure;

[0031] Figure 15 shows a flowchart of the method for converting low-bit-width data to high-bit-width data; and

[0032] Figures 16a to 16c exemplarily illustrate the data transformation when converting FP6 to FP8. Detailed Implementation

[0033] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0034] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. "First," "second," "third," and "fourth," etc., also do not necessarily mean only one, but can also mean multiple. The terms "comprising" and "including" used in the specification and claims of this first disclosure indicate the presence of the described feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof.

[0035] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0036] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0037] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0038] Modern semiconductor manufacturing begins with a single, complete wafer, a thin, circular sheet of pure silicon, typically available in sizes such as 6 inches, 8 inches, and 12 inches. The wafer is then diced into smaller pieces called dies. Each die contains a chip with its wiring installed to perform a specific electrical function. These dies are then packaged into a single chip. The purpose of packaging is to house, secure, seal, and protect the chip, as well as enhance its electrical and thermal performance. Simultaneously, wires connect the chip's contacts to the lead pins of the package casing, thus completing a chip package structure.

[0039] Memory is used to temporarily store the computational data required by the on-chip system, as well as data exchanged with external memory. In this embodiment, the memory can be high-bandwidth memory (HBM), a high-performance DRAM fabricated using 3D stacking technology, suitable for applications with high memory bandwidth requirements, such as graphics processors, network switching and forwarding devices (e.g., routers, switches).

[0040] A System-on-a-Chip (SoC) refers to a technology that integrates a complete system onto a single chip, grouping all or part of the necessary electronic circuitry. In this embodiment, the SoC is mounted on a board. Figure 1 shows a schematic diagram of the structure of a board 10 according to an embodiment of this disclosure. As shown in Figure 1, the board 10 includes a combined processing device 101, which is an artificial intelligence computing unit used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant characteristic of cloud intelligence applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. The board 10 of this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and abundant computing power.

[0041] The combined processing unit 101 is connected to an external device 103 via an external interface device 102. The external device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from the external device 103 to the combined processing unit 101 via the external interface device 102. The calculation results of the combined processing unit 101 can be transmitted back to the external device 103 via the external interface device 102. Depending on the application scenario, the external interface device 102 may have different interface types, such as a PCIe interface.

[0042] The board 10 also includes an external memory 104 for storing data, which includes one or more memory cells 105. The external memory 104 is connected to and transmits data with the controller 106 and the combined processing unit 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the combined processing unit 101. For this purpose, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0043] Figure 2 is a schematic diagram illustrating the combined processing apparatus 101 of this embodiment. As shown in Figure 2, the combined processing apparatus 101 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204. In one application scenario, the computing device 201, the interface device 202, and the processing device 203 are integrated into the aforementioned system-on-a-chip. In another application scenario, the computing device 201 itself is the aforementioned system-on-a-chip.

[0044] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0045] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0046] Processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 can be one or more types of processors, such as a central processing unit, graphics processing unit, or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, for the purposes of this disclosure only, computing device 201 can be considered to have a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0047] DRAM204 is the aforementioned high-bandwidth memory used to store data to be processed. Its size is typically 16G or larger, and it is used to store data of computing device 201 and / or processing device 203.

[0048] Figure 3 shows a schematic diagram of the internal structure of the computing device 201. The computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design, which includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0049] There can be multiple external storage controllers 301; two are shown exemplarily in the figure. These controllers respond to access requests from the processor core to access external storage devices, such as DRAM 204 in Figure 2, thereby reading data from or writing data to external storage. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controllers 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between the modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201; four are shown exemplarily in the figure. With hardware development, the computing device 201 disclosed herein may also include eight, sixteen, sixty-four, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0050] Each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0051] Four processor cores 306 are shown exemplarily in the figure, but this disclosure does not limit the number of processor cores 306. Their internal architecture is shown in Figure 4a. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0052] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0053] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0054] Storage module 43 is used to store or move related data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 is used to store input, output data and intermediate results for the processor core 306 to calculate; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access of NRAM 431 / WRAM 432 and DRAM 204 through broadcast bus 309; MVDMA 434 controls the memory access of NRAM 431 / WRAM 432 and SRAM 308.

[0055] Returning to Figure 3, storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between cluster 305 and DRAM 204, communication between clusters 305, and communication between processor cores 306. In other embodiments, storage core 307 has scalar operation capabilities for performing scalar operations.

[0056] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0057] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between cluster 305 and DRAM 204, respectively. These will be explained below.

[0058] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0059] CDMA 310 is used to control memory accesses of SRAM 308 between different clusters 305 within the same computing device 201. GDMA 311 works in conjunction with external memory controller 301 to control memory accesses from SRAM 308 of cluster 305 to DRAM 204, or to read data from DRAM 204 into SRAM 308.

[0060] Before providing a detailed description of the technical solutions disclosed herein, we will first introduce several terms involved in this disclosure.

[0061] Floating-point numbers: The IEEE floating-point standard uses V = (-1). sign *mantissa*2 E A number is represented in the form of . Here, `sign` is the sign bit, where 0 represents a positive number and 1 represents a negative number; `E` represents the exponent (or index), which is a weighted sum of 2 raised to the power of `E` (which may be a negative power); `mantissa` represents the mantissa, a binary fraction. Floating-point numbers are represented in computers using three fields, each encoded separately:

[0062] (1) A single sign bit directly encodes the symbol s;

[0063] (2) The exponent field of the k-bit is encoded as exp=e(k-1)......e(1)e(0);

[0064] (3) The n-digit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent field is all 0.

[0065] Fixed-point numbers are a way of representing numbers in computers, characterized by a fixed decimal point. Fixed-point numbers can represent integers or decimals, depending on the implicit position of the decimal point. A fixed-point number consists of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision.

[0066] Bit width: How many bits are used to represent data.

[0067] Quantization: The process of converting high-precision numbers into lower-precision data that takes up less memory.

[0068] Dequantization: The process of converting low-precision numbers into higher-precision data.

[0069] FP*: Represents a floating-point data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0070] BF16 and TF32 are data types suitable for deep learning, which improve computational performance by sacrificing precision and range.

[0071] int*: Represents an integer data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0072] ExMy: A data format where E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width.

[0073] MX: refers to the Microscaling specification (see “OCP Microscaling Formats (MX) Specification Version 1.0”).

[0074] Extended MX: refers to a quantization scheme proposed in the embodiments disclosed herein.

[0075] Granular quantization: This includes MX and extended MX, as well as other quantizations with similar data structures. The quantization granularity is determined based on the amount of data that the arithmetic unit can process at one time.

[0076] It should be noted that in this disclosed embodiment, the data that needs to be quantized or dequantized can be various types of data in the neural network, including but not limited to input neurons, weights, output neurons, gradients, etc. The neural network provided in this disclosed embodiment can be applied to various fields and can process various types of data. For example, in image recognition and processing, facial recognition, object detection, image classification, and medical image analysis can include data types such as pixel data and image features; in natural language processing (NLP), language translation, sentiment analysis, text summarization, and speech recognition can include data types such as text data, word embeddings, and sentence structure; in speech recognition and processing, intelligent assistants, automatic caption generation, and speech-to-text conversion can include data types such as audio signals and spectrograms; in recommendation systems, personalized content recommendation, product recommendation, and advertising can include data types such as user behavior data, item features, and rating data; in healthcare, disease diagnosis, drug discovery, and gene sequence analysis can include data types such as medical records, biomarker data, and genomic data; in the financial field, risk assessment, fraud detection, and stock market prediction can include data types such as transaction data, user credit scores, and market data; and in autonomous driving, vehicle perception, decision-making, and path planning can include data types such as sensor data, environmental features, and traffic signals. In gaming and entertainment, AI applications such as game AI, virtual reality, and animation generation can include data types such as game state data and user interaction data. In scientific research, applications such as physics simulation, chemical compound prediction, and astrophysical analysis can include data types such as experimental data, simulation results, and observational data. Generative AI applications, on the other hand, utilize complex algorithms, models, and rules to learn from large-scale datasets to create new, original content. This includes, but is not limited to, creating various types of content such as text, images, audio, video, and code. Correspondingly, the input neuron data processed can be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc., and the corresponding output neuron data can also be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc. The output of the neural network can include a probability score of an image belonging to a specific object category, a probability score of a document relating to a specific topic, a probability score of the correct translation of a text fragment in the target language from a text fragment in the source language, or a probability score of the correct transcription of a text fragment from spoken language, etc. These output data can also be quantized or unquantized. By quantizing / dequantizing the data, the processing efficiency of these tasks can be maintained while reducing the computational load and saving computing resources.The data quantization scheme of this disclosure embodiment can also be used in the inference and training of neural networks. By performing the quantization process described in this disclosure embodiment on the data of the neural network, the accuracy of the task can be guaranteed when the neural network performs one or more of the aforementioned processing tasks, while improving the speed and performance of the neural network model's inference and training.

[0077] Exemplary MX Quantization Scheme

[0078] Figure 4b illustrates a schematic diagram of the MX quantized data format. As shown, the MX quantized data format is characterized by three parts: the scaling factor (X, shared scale), the scaling block size (K, scaling block size), and the private elements within the block (P). i ).

[0079] The scaling factor X relates to the data type or encoding method of the scaling factor. Private element (P) i It also involves its data type or encoding method. The scaling block size (K) represents the private element P within the block. i The number of elements P within the block. i All elements have the same data type and therefore the same bit width. The scaling factor X is shared among these K elements. The data type of the elements and the data type of the scaling factor can be chosen independently. In a sense, MX can be viewed as a mechanism for constructing vector data types based on scalar data types. The values ​​V1,…,V represented in an MX block are... K It can be determined as follows: V i =XP i .

[0080] As further illustrated in Figure 4b, W represents the number of bits used to encode the shared scaling factor X, i.e., the bit width of the scaling factor X; d represents the number of bits used to represent each element P. i The number of bits, that is, the element P i The bit width is . Therefore, for an MX block containing K elements, it can be encoded using (W+Kd) bits.

[0081] Table 1 shows some common MX formats and related parameters. Table 1

[0082] In Table 1 above, ExMy represents a scalar format notation with x exponent bits and y mantissa bits. For example, FP8 (E4M3) represents an FP8 format with 1 sign bit, 4 indicator bits, and 3 mantissa bits. When y is 0 (e.g., E8M0), the format does not include a sign bit.

[0083] Furthermore, in MX quantization, the scaling factor X is calculated as follows:

[0084] (1) Determine the number of elements P that are less than or equal to K. i The largest power of the absolute value in the range;

[0085] (2) Divide the largest power of 2 by element P. i The data type can represent the maximum power of 2, and the calculated result is set as the scaling factor Scale X.

[0086] The inventors have discovered that, for MX quantization, processing the original data into ultra-low precision data types such as F4 (representing a 4-bit data format) leads to a sharp decrease in the accuracy of large models, significantly impacting the accuracy of the computational results. Taking image recognition tasks as an example, task accuracy or processing performance is reflected in the accuracy of the image recognition results, i.e., whether the image recognition computation is accurate. The quantized image data obtained after quantization using the aforementioned quantization method affects the accuracy of the image recognition results.

[0087] The inventors also discovered that if the data type of the scaling factor used in the quantization method is the same as the original data type, although the above-mentioned problem of decreased accuracy can be solved to some extent, this method will lead to greater resource overhead (such as storage space, area overhead required for hardware support, etc.).

[0088] Exemplary Extended MX Quantization Method

[0089] Based on the above findings, this disclosure provides a novel data quantization scheme. This scheme determines a scaling factor based on the maximum absolute value of the data to be quantized, and obtains a scaling coefficient by converting the scaling factor to a different data type. Since the converted scaling coefficient has a non-zero mantissa, the accuracy of the quantized data is improved, resulting in processing tasks performed on the quantized data according to the embodiments of this disclosure exhibiting better accuracy. Furthermore, setting the quantization granularity based on the hardware's single-processing capacity facilitates the quantization of a block of granularity in a single hardware operation. This makes online quantization / dequantization more convenient, facilitating pipelined design and enabling neural network models using this hardware as a computing platform to perform quantization operations at various stages, including inference and training, thereby improving the computational efficiency of the neural network model.

[0090] In the extended MX quantization scheme, the scaling factor is calculated as follows:

[0091] (1) Determine the maximum absolute value in a set of data to be quantified;

[0092] (2) Obtain the maximum value that the target data type can represent after the data to be quantized is quantized;

[0093] (3) Determine the scaling factor corresponding to this set of data to be quantized based on the ratio between the maximum absolute value and the maximum value;

[0094] (4) Convert the scaling factor from the first data type to the second data type to obtain the scaling coefficient, wherein the bit width of the second data type can be smaller than the bit width of the first data type.

[0095] The target data type is the data type of the quantized data obtained after quantizing the data to be quantized. Before quantization, the target data type can be set as needed, and its precision is generally lower than that of the original data type of the data to be quantized. In some embodiments, the original data type of the data to be quantized may include at least one of FP16, BF16, FP32, FP64, TF32, etc.; the target data type may include at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, etc. Among these, floating-point data types such as FP8, FP6, and FP4 can support multiple data formats. Taking FP4 as an example, it can be divided in several ways according to the rule of "sign bit + exponent bit + mantissa bit," with the sign bit typically occupying 1 bit. Common combinations of exponent and mantissa bits include E3M0 and E2M1. Similarly, FP8 can support data formats such as E4M3 and E5M2, and FP6 can support data formats such as E3M2 and E2M3.

[0096] Due to the bit width limitations of each data type, each data type has a maximum representable value. For example, the maximum representable value when the target data type is FP4 is 6; the maximum representable value when the target data type is int8 is 127. Typically, there is a one-to-one correspondence between the maximum value and the target data type, which can be obtained through methods such as table lookup.

[0097] Since the data type of the absolute maximum value is the same as the original data type of the data to be quantized, the data type of the scaling factor obtained by the ratio (i.e., the first data type) is the same as the original data type of the corresponding data to be quantized, or the bit width of the first data type can be greater than the bit width of the original data type.

[0098] The first data type is the data type of the scaling factor, and the second data type is the data type of the scaling coefficient. Through data type conversion, the bit width of the second data type is made smaller than that of the first data type, so that the precision of the scaling coefficient is less than that of the scaling factor. In some embodiments, converting the scaling factor to the scaling coefficient can be achieved through a conversion instruction, such as cvt_to_scale_dtype(). In other embodiments, converting the scaling factor to the scaling coefficient supports multiple round modes, such as rn (round to nearest, ties to even).

[0099] In other embodiments, the second data type can be expressed as ExMy, where E represents the exponent bit, M represents the mantissa bit, x represents the bit width of the exponent bit, y represents the bit width of the mantissa bit, and x + y < the bit width of the original data type. For example, assuming the original data type is FP16, then x + y < 16. The specific values of x and y can be set according to requirements or the hardware environment, x is a positive integer, and y ≥ 0. In still other embodiments, 3 ≤ x + y < the bit width of the original data type. In some embodiments, 8 ≤ x + y < the bit width of the original data type. In some embodiments, the second data type may include at least one of, for example, E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, E3M9, etc. In some preferred embodiments, y > 0. In further preferred embodiments, the second data type may be E8M2 or E8M3.

[0100] For the sake of easy understanding, still taking the original data type as FP16 as an example, when 8 < x + y < the bit width of the original data type, the bit width (i.e., x + y) represented by the second data type ExMy of the scaling coefficient can be between 8 bits and 16 bits. Using such a scaling coefficient to perform quantization processing on the operation data of the neural network and using the quantized data obtained for inference and training of the neural network, etc., the precision of the neural network obtained is between the 8-bit representation (such as E8M0 represented in MX) and the original 16-bit representation (such as E5M10, E8M7, that is, FP16, BF16). Compared with the MX format, such a scaling coefficient disclosed in the present disclosure is beneficial to improving the precision of the quantized data; compared with the case where the data type of the scaling coefficient is the same as the original data type, such a scaling coefficient disclosed in the present disclosure is beneficial to reducing hardware resource overhead or storage overhead, etc.

[0101] Furthermore, since the mantissa bit width affects the quantization accuracy, a larger mantissa bit width can provide a scaling factor with higher accuracy, thereby preserving more original data details during the quantization process. Therefore, compared with the scaling factor of E8M0 format in MX quantization, when the mantissa bit width y of the scaling factor in this disclosed embodiment is greater than 0, it is beneficial to improve the accuracy of quantized data and the accuracy of processing tasks.

[0102] Furthermore, this disclosed embodiment enables the scaling factor to support multiple data types by performing data type conversion, thereby allowing for more flexible adaptation to different data characteristics and precision requirements. Regarding the selection of the exponent and mantissa bit widths of the scaling factor, the distribution characteristics of the data to be quantized can be considered. A larger exponent bit width allows for the representation of more values. If the range of values ​​in a set of data to be quantized varies significantly, the exponent bit width of the scaling factor can be set larger; conversely, if the range of values ​​in a set of data to be quantized varies only slightly, the exponent bit width of the scaling factor can be set smaller, while the mantissa bit width can be set larger.

[0103] For example, suppose the second data type needs to be set to 8 bits. If the absolute value range of a set of data to be quantized involves 2... -127 ~2 127 Then the second data type can be set to E8M0; if the absolute value range of a set of data to be quantized involves 2 -4 ~2 4 Then the second data type can be set to E3M5. Of course, it is understandable that the bit width of the second data type is not limited to 8 bits, and can also be set to 5 bits, 4 bits, etc., as needed.

[0104] After determining the scaling factor in the above manner, the scaling factor can be used to quantize the corresponding data to be quantized, or to dequantize the quantized data.

[0105] In some embodiments, quantization processing may include: dividing the data to be quantized by a scaling factor to obtain an intermediate quantization result; and performing data type conversion on the intermediate quantization result to obtain quantized data with the target data type. This data type conversion can also be implemented using a conversion instruction, such as `cvt_to_scale_dtype()`. This conversion instruction can be a hardware or software instruction at various levels, such as processor microinstructions, ISA instructions, or function instructions implemented in high-level languages ​​at various software levels.

[0106] Dequantization is generally the process of restoring low-precision data to high-precision data. In some embodiments, dequantizing quantized data using a scaling factor may include multiplying the quantized data with the scaling factor. In other embodiments, the quantized data may be converted from a target data type to a desired data type before being multiplied with the scaling factor, wherein the precision of the desired data type is higher than that of the target data type. In still other embodiments, dequantization can restore the quantized data to its original state before the quantization operation.

[0107] As can be seen from the second data type of the scaling factor obtained from the previous conversion, the bit width of a single scaling factor is 3 to 15 bits. For example, E8M4 requires 12 bits, which typically requires 2 bytes (2B) of storage space, so it will definitely not be fully utilized. This inevitably leads to a waste of storage space and transmission bandwidth. To solve this problem, in some embodiments, two or more scaling factors can be compressed, thereby saving storage space and bandwidth.

[0108] In some embodiments, after quantizing two or more sets of data to be quantized using the aforementioned method to obtain two or more scaling factors, these two or more scaling factors can be compressed. Then, the compressed scaling factors are used to quantize the corresponding sets of data to be quantized, respectively, to output quantized data with the target data type.

[0109] Specifically, after obtaining the scaling factors as described above, a compression process can be included, which specifically includes: determining the shared exponent of the Z scaling factors to be compressed, where Z ≥ 2; determining the Z new mantissas corresponding to these Z scaling factors; and bundling the shared exponent with the Z new mantissas for output for association. That is, after compression, the exponents of the Z scaling factors are reused, and their mantissas are different, thereby saving at least Z-1 exponent bits.

[0110] In some implementations, determining the shared exponent for the Z scaling factors to be compressed may involve setting the largest exponent among these Z scaling factors as the shared exponent. Choosing the largest exponent as the shared exponent ensures the precision of larger numbers as much as possible, since larger numbers typically have a greater impact on the calculation results.

[0111] In some implementations, determining the Z new mantissas corresponding to the Z scaling factors can include: setting the new mantissa of scaling factors with an exponent less than the maximum exponent to 0; and setting the new mantissa of scaling factors with an exponent equal to the maximum exponent to the original mantissa. It can be understood that for scaling factors with an exponent equal to the maximum exponent, since the exponent remains unchanged, its mantissa remains unchanged; however, for scaling factors with an exponent less than the maximum exponent, after the exponent is scaled up to the maximum exponent, to maintain equivalence, the mantissa would be proportionally reduced, exceeding the range of values ​​that the mantissa can represent. Therefore, its new mantissa is set to 0 here. This setting will result in some precision loss, but compared to the storage and bandwidth gains, this precision loss is acceptable in some cases.

[0112] In some implementations, the shared exponent and the Z new mantissas can be bundled together in sequence for association. For example, first the shared exponent, then the new mantissa of the first scaling factor, the new mantissa of the second scaling factor, and so on up to the new mantissa of the Z scaling factor.

[0113] Next, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized, so as to output quantized data with the target data type. Since the exponent and mantissa of the scaling factors with an exponent smaller than the maximum exponent will change after compression, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized separately in order to maintain the consistency of quantization and dequantization.

[0114] In some implementations, the second data type of the Z scaling factors to be compressed is the same. For example, all Z scaling factors are of type E8M4. In this way, the shared exponent bit width after compression is 8 bits, and the Z new mantissas are 4Z bits.

[0115] In some implementations, the Z scaling factors belong to Z adjacent groups of data to be quantized. This allows the data to be processed sequentially without requiring additional information to indicate the scaling factors that are compressed together.

[0116] In some implementations, Z can be 2, 4, 8, or other values. Those skilled in the art can choose an appropriate Z value based on the number K of data to be quantized in a set of data. For example, when K is small, the value of Z can be larger; when K is large, the value of Z needs to be smaller, thereby ensuring that the benefit of using the compression scheme is greater than the benefit of amplifying K by Z times.

[0117] Similarly, when dequantizing extended MX quantized data using a compressed format, the scaling factors need to be decompressed first. Specifically, during dequantization, in response to the scaling factors being in a compressed format, the compressed scaling factors are decompressed; and the decompressed scaling factors are used to dequantize the quantized data.

[0118] Based on the compression method and bundling approach described above, the bundled Z scaling factors can be decoded into Z independent scaling factors. Specifically, the shared exponent can be combined with each of the Z new mantissas to obtain the Z decompressed scaling factors.

[0119] It is understood that in some embodiments, in addition to calculating and storing the scaling factor, the reciprocal of the scaling factor is also calculated and stored. This facilitates converting the division operation between the data to be quantized and the scaling factor into a multiplication operation between the data to be quantized and the reciprocal of the scaling factor during the quantization process. The reciprocal of the scaling factor has the same data type as the scaling factor. Therefore, the compression and decompression processes described above can also be applied to the reciprocal of the scaling factor, which will not be described in detail here.

[0120] In the extended MX quantization scheme described above, the scaling factor is determined based on the maximum absolute value of the data to be quantized. This ensures the accuracy of the data after quantization based on the maximum absolute value. Furthermore, since the larger the value of the data to be quantized during the quantization process, the greater the loss of accuracy and the greater the impact on the task calculation results, ensuring the accuracy of the data based on the maximum absolute value helps to reduce the loss of task accuracy. Therefore, compared to methods such as calculating the scaling factor in MX, the extended MX quantization scheme is better at ensuring task accuracy.

[0121] Furthermore, compared to the quantization method where the data type of the scaling factor is the same as the original data type, the scaling factor with a second data type obtained by the number of revolutions in this disclosed embodiment can reduce the storage space required to store the scaling factor and the hardware overhead required to execute the task while ensuring the accuracy of the task.

[0122] In some embodiments, the number of data to be quantized in a set of data to be quantized can be determined based on the single processing volume and / or the number of input ports of the processing circuit performing the quantization process; or the number of data to be quantized in a set of data to be quantized can be determined based on the single operation volume and / or the number of input ports of the arithmetic unit that performs the operation using the quantized data.

[0123] For example, the number of data points to be quantized in a set of data can be determined based on the processing capacity of the processing circuit (i.e., the amount of data that can be processed in a single run), or based on the processing capacity of the arithmetic unit that performs operations such as multiply-accumulate or vector inner product using quantized data (i.e., the amount of data that can be processed in a single run), and grouped according to this granularity. With this setup, a set of data points sharing a single scaling factor can be processed in one computation cycle, without needing to span multiple computation cycles, thus improving hardware implementation and fully utilizing hardware computing power.

[0124] For example, the number of data points to be quantized in a set of data can be determined based on the number of input ports of the processing circuit performing quantization or the number of input ports of the arithmetic unit that uses quantized data for calculation, and grouped accordingly. With this setup, a set of data to be quantized can be input at once using the number of hardware input ports, allowing a set of data sharing a single scaling factor to be processed in a single input, eliminating the need for multiple inputs. This improves hardware implementation, reduces hardware processing latency, and lowers I / O overhead.

[0125] For example, assuming the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is also 32, then the data points to be quantized can be grouped into sets of 32. If the aforementioned processing circuit or arithmetic unit can process 64 data points in a single operation, and its corresponding number of input ports is less than 64, for example, 32, then the data points to be quantized can be grouped according to the number of input ports, ensuring that the number of data points to be quantized in each group is equal to the number of input ports. In this case, 32 data points are input at once, and the two groups of data can be processed in parallel; alternatively, the 64 data points to be quantized can still be grouped together and input twice to complete the processing of one group of data points at once. If the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is more than 32, then the 32 data points to be quantized can be grouped together to achieve one-time input and processing of one group of data points.

[0126] In some embodiments, the data to be quantized can be elements of tensor data. Dividing multiple data to be quantized into multiple groups can include: dividing the multiple data to be quantized into multiple groups along at least one dimension of the tensor data. For example, suppose tensor data A(B, S), where B and S represent the dimensions of the first and second dimensions, respectively. If the division is performed along the second dimension, taking a grouping granularity of 32 as an example (i.e., each group contains 32 data to be quantized), dividing 32 consecutive data to be quantized along dimension S into a group results in (B×S / 32) groups. Taking each group as a data block, the shape of the tensor composed of each data block after grouping (i.e., the dimensions and dimensions of the tensor) can be represented as (B, S / 32), that is, the dimension of the second dimension after grouping is S / 32.

[0127] For example, suppose tensor data A'(B, S, D), where B, S, and D represent the dimensions of the first, second, and third dimensions, respectively. If we divide the data along the third dimension, taking a grouping granularity of 64 as an example, we can divide 64 consecutive data points to be quantized along the third dimension into a group, resulting in (B×S×D / 64) groups. Each group can be considered as a data block. The shape of the tensor composed of these data blocks after grouping can be represented as (B, S, D / 64), which means the size of the third dimension after grouping is D / 64.

[0128] It is understandable that dividing multiple data sets to be quantized into multiple groups is not limited to dividing them only along one dimension of the tensor data; it can also be done along multiple dimensions as needed. The grouping granularity is not limited to the aforementioned 32 or 64; it can be set as needed, for example, 16, 128, etc. The grouping granularity is related to the computing power or input port limitations of the processing circuit / arithmetic unit mentioned earlier, and will not be elaborated upon here.

[0129] It should be further explained that as the number of groups (i.e., the number of data blocks) increases, the number of scaling factors also increases accordingly. The scaling factors can represent a larger number of data points, which is beneficial for further improving data quantization accuracy. In particular, as the number of scaling factors increases, the sum of the exponent bit width and mantissa bit width increases, thus allowing for the representation of more data points than the E8M0 data format in MX quantization. Consequently, compared to MX quantization, the data accuracy represented by the scaling factors in the extended MX quantization of this disclosed embodiment is higher.

[0130] Therefore, based on the technical solution of obtaining scaling factors through data type conversion and the maximum absolute value, group quantization is used to increase the quantization granularity. A corresponding scaling factor is provided for each group of data to be quantized, which is conducive to further improving the accuracy of the quantized data and the accuracy of the executed task.

[0131] Exemplary device for extended MX quantization

[0132] Figure 5 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure. As shown, the apparatus 500 may include a first processing circuit 510 and a second processing circuit 520, wherein the first processing circuit 510 is used to output a scaling factor, and the second processing circuit 520 is used to perform quantization processing. Specifically, the first processing circuit 510 may be used to: determine the maximum absolute value in a set of data to be quantized; obtain the maximum value that the target data type can represent after quantization of the data to be quantized; determine a scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling factor, wherein the bit width of the second data type is smaller than the bit width of the first data type. The second processing circuit 520 may be used to quantize a set of data to be quantized using the scaling factor to output quantized data with the target data type.

[0133] In some embodiments, the first processing circuit 510 may be further configured to: determine the number of data to be quantized in a set of data to be quantized based on the single processing volume and / or the number of input ports of the second processing circuit 520; or determine the number of data to be quantized in a set of data to be quantized based on the single operation volume and / or the number of input ports of the arithmetic unit that performs operations using quantized data.

[0134] In some embodiments, the apparatus 500 may further include a compression circuit 540. The compression circuit 540 is configured to compress two or more scaling factors after quantization of two or more sets of data to be quantized. In this case, the second processing circuit 520 is configured to quantize the corresponding sets of data to be quantized using the compressed scaling factors, respectively, to output quantized data with the target data type.

[0135] Correspondingly, when device 500 needs to dequantize extended MX quantized data using a compressed format, the scaling factors must first be decompressed. In these embodiments, device 500 may also include a decompression circuit (not shown in the figure). The decompression circuit is configured to decompress the compressed scaling factors in response to the scaling factors being in a compressed format. At this time, the circuit in device 500 that performs the dequantization process (e.g., the second processing circuit 520 or the third processing circuit 530) can be configured to perform dequantization processing on the quantized data using the decompressed scaling factors.

[0136] The operations performed by the first processing circuit 510, the second processing circuit 520, the compression circuit 540, and the decompression circuit have been described in detail above in conjunction with the quantization method, and will not be repeated here.

[0137] In some embodiments, the first processing circuit 510 may include: a comparison circuit 511, which can be used to determine the maximum absolute value in a set of data to be quantized; a first division circuit 512, which can be used to obtain the maximum value that the target data type after quantization of the data to be quantized can represent, and determine the scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and a first rotation circuit 513, which can be used to convert the scaling factor from a first data type to a second data type to output the scaling coefficient. In some embodiments, the first division circuit 512 can obtain the maximum value by means of table lookup or other methods.

[0138] In some embodiments, the second processing circuit 520 may include a second division circuit 521 and a second rotation circuit 522, wherein the second division circuit 521 can be used to perform a division operation on the data to be quantized and the scaling factor, and the second rotation circuit 522 can be used to perform a rotation operation on the result of the division operation (i.e., the intermediate quantization result) to output the result of quantization of the data to be quantized (i.e., the quantized data).

[0139] In other embodiments, the second processing circuit 520 can also be used to perform dequantization processing on the quantized data using scaling factors. For example, the second processing circuit 520 may also include a multiplication circuit for multiplying the quantized data and the scaling factors. In still other embodiments, the second rotation circuit 522 can also be used to perform rotation operations on the quantized data to convert the quantized data from the target data type to the data type required for dequantization processing; the multiplication circuit is used to multiply the rotated quantized data and the scaling factors to obtain the dequantization result. When the required data type is the original data type of the aforementioned data to be quantized, dequantization processing can restore the quantized data to the data to be quantized before the quantization operation.

[0140] It is understood that the dequantization process is not limited to being performed by the second processing circuit 520 as described above, but can also be performed by other circuits. For example, in some embodiments, the apparatus 500 may further include a third processing circuit 530 (shown in a dashed box in the figure) for dequantizing the quantized data using a scaling factor. The third processing circuit 530 may receive, for example, quantized data output by the second processing circuit 520, and may receive scaling factors output from the first processing circuit 510. In some embodiments, a storage circuit may be further included between the third processing circuit 530 and the second processing circuit 520, or the second processing circuit 520 may include a storage circuit for storing quantized data so that the third processing circuit 530 can read the quantized data from the storage circuit for dequantization processing.

[0141] In some embodiments, when a piece of data to be quantized is input to the comparison circuit 511, the comparison circuit 511 can output the absolute value of that piece of data as the maximum absolute value. In some embodiments, when multiple pieces of data to be quantized are input to the comparison circuit 511, the comparison circuit 511 can output the maximum absolute value among the multiple pieces of data to be quantized. In still some embodiments, the comparison circuit 511 can also be used to divide the multiple pieces of data to be quantized into multiple groups and output the maximum absolute value of each group of data to be quantized; the first division circuit 512 can be further used to determine a scaling factor corresponding to each group of data to be quantized; and the first revolution circuit 513 can be further used to determine a scaling coefficient corresponding to each group of data to be quantized based on each scaling factor. In some embodiments, the data to be quantized is an element of tensor data, and the comparison circuit 511 can be further used to divide the multiple pieces of data to be quantized of the tensor data into multiple groups in at least one dimension of the tensor data.

[0142] It is understood that the above description is exemplary and not restrictive. For example, the first processing circuit for obtaining the scaling factor may not be limited to the comparison circuit, the first division circuit, and the first rotation circuit described above, but may also be implemented using, for example, a lookup table (circuit). Since the dividend (the maximum value that the target data type can represent) is a constant, a lookup table or similar method can be used to directly determine the value based on the input data, thereby implementing the division operation and data type conversion functions described above to obtain the scaling factor. Furthermore, to facilitate understanding of the grouping function implemented by the comparison circuit, an exemplary description will be provided below with reference to Figure 6.

[0143] Figure 6 shows a schematic block diagram of a comparison circuit according to some embodiments of this disclosure. As shown, the comparison circuit 511 may include a comparator tree composed of multiple levels of comparators, for example, it may include layers a, b, c, ..., z, each level including one or more comparators, with the number of comparators decreasing sequentially at each level. The outputs of multiple comparators in the upper level of an adjacent level can be input to the comparators in the lower level. In practical applications, the outputs of the required levels can be selected as needed by means of software or hardware to achieve grouping of multiple data to be quantized and to obtain the maximum absolute value of each group of data to be quantized. For ease of understanding, further examples will be provided below with reference to Figure 3.

[0144] As further illustrated in Figure 6, layer a includes comparators a1, a2, a3, ..., an, where n represents the number of comparators in layer a; layer b includes comparators b1, b2, etc.; layer c includes comparator c1, etc.; and layer z includes comparator z1. It is understood that the comparator tree shown in the figure is exemplary, and the number of levels in the comparator tree can be set more or less as needed, and the number of comparators included in each level can also be set as required.

[0145] In some embodiments, the number of comparators *n* in layer *a* can be the same as the number of input data to be quantized, so that each data point to be quantized is input into a corresponding comparator in layer *a*. As the number of comparators in subsequent layers decreases sequentially, the output can be gradually reduced to obtain the desired grouping result and the maximum absolute value of each group. For example, assuming the input layer receives 128 data points to be quantized, and layer *a* has 128 comparators, then each data point to be quantized is input into a corresponding comparator in layer *a*. The output of layer *a* can then divide the 128 data points to be quantized into 128 groups. Assuming layer *b* includes 64 comparators, then the output of every two comparators in layer *a* can be input into a comparator in the next layer, layer *b*. For example, the outputs of comparators *a1* and *a2* in layer *a* can be input into comparator *b1* in layer *b*. Layer *b* then outputs 64 maximum absolute values, thus dividing the 128 data points to be quantized into 64 groups. Assuming layer c contains 32 comparators, the outputs of every two comparators in layer b can be input into a comparator in the next layer c. For example, the outputs of comparators b1 and b2 in layer b can be input into comparator c1 in layer c. At this point, layer c outputs 32 maximum absolute values, effectively dividing the 128 data points to be quantized into 32 groups. This process continues until the last layer z outputs a single maximum absolute value, thus dividing the 128 data points to be quantized into one group.

[0146] Taking this embodiment as an example, in practical applications, assuming that 128 data points to be quantized need to be divided into groups of four, resulting in 32 groups, the output of layer C can be selected as the output of comparator circuit 511 through methods such as programming, logic control, or selectors. This allows the first division circuit to determine the scaling factor corresponding to each of the 32 groups of data to be quantized based on the 32 outputs of layer C (i.e., the 32 maximum absolute values). Therefore, the comparator tree provides multiple granularity grouping methods, allowing users to flexibly select the output of a certain level in the comparator tree according to the required quantization granularity.

[0147] Furthermore, it should be noted that the comparator tree structure is not limited to the two comparators in each level shown in the diagram whose outputs correspond to one comparator in the next level. It can also be configured to have, for example, three or more comparators whose outputs correspond to one comparator in the next level. For instance, assuming level a has 128 comparators, level b can have 4 comparators. That is, the outputs of the 32 comparators in level a correspond to one comparator in level b. Therefore, the output of level b can divide the 128 data points to be quantized into 4 groups and output the maximum absolute value of the 32 data points in each group.

[0148] In other embodiments, the number of comparators n in layer a can be different from the number of input data to be quantized, as illustrated below. Assuming the number of comparators in layer a is less than the number of data to be quantized, multiple data to be quantized can be input into a single comparator in layer a. This achieves the division of multiple data to be quantized into fewer groups for output in layer a. In this embodiment, the data to be quantized input to each comparator in layer a can also be controlled before layer a (e.g., in the input layer) by using methods such as address decoders, multiplexers (MUX), or other logic circuits.

[0149] For example, suppose the input layer receives 128 data points to be quantized, and layer 'a' has 64 comparators. Then, every two data points to be quantized are input into one of the corresponding comparators in layer 'a'. At this point, the output of layer 'a' divides the 128 data points into 64 groups. The number of comparators in subsequent layers is halved sequentially until the last layer outputs the maximum absolute value, at which point the 128 data points are divided into one group.

[0150] The foregoing description, in conjunction with Figure 6, exemplarily illustrates the scheme for grouping and outputting the maximum absolute value using a comparison circuit according to an embodiment of this disclosure. It is understood that the structure of the comparison circuit shown in the figure and the preceding description are exemplary and not restrictive. For example, the last level of the comparison circuit 511 may not be limited to including only one comparator as shown in the figure, but may also include multiple comparators as needed, such as using layer c in the figure as the last level. Furthermore, the comparator circuit 511 may not be limited to including only one comparator tree, but may also include multiple comparator trees as needed, thereby enabling, for example, parallel grouping. The comparison circuit of this embodiment of the disclosure may not be limited to being implemented using the aforementioned comparator tree, but may also be implemented based on radix sorting or other methods.

[0151] It is also understood that the grouping method in this disclosed embodiment is not limited to the comparison circuit, but can also be implemented by combining software. For example, by setting grouping parameters (including information such as the number of data to be quantized in each group K) through software, the comparison circuit can read K consecutive data to be quantized as a group according to these grouping parameters, and directly output the maximum absolute value in each group of data to be quantized.

[0152] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0153] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs (Graphics Processing Units), FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM (read-only memory), and RAM (random access memory), etc.

[0154] In this disclosed embodiment, when referring to "particle quantization format," "particle size format," or "particle quantization," it refers to the three-part data structure shown in Figure 1. However, the specific element data types, scaling block sizes, scaling factor data types, etc., are not limited to those listed in Table 4b, but can include many other combinations. In other words, the particle quantization format can include the MX quantization format proposed by OCP, the extended MX quantization format proposed in this disclosed embodiment, or any other data format conforming to the structure shown in Figure 1.

[0155] In this disclosed embodiment, the term "instruction" can include software instructions, hardware instructions, firmware instructions, or any combination thereof. Software instructions typically refer to statements or commands in a programming language; they are high-level, abstract instructions, such as function calls, machine code, and bytecode. Hardware instructions, also known as machine instructions or instruction set architecture (ISA) instructions, are low-level commands that computer hardware can directly recognize and execute, such as processor instructions and CPU instructions. Firmware instructions include, for example, opcodes and microcode. Furthermore, different CPU architectures and different instruction sets can have different hardware instruction sets. Based on the complexity of the computer instruction set, hardware design, execution speed, compiler complexity, and instruction format, they can be further classified as: CISC (Complex Instruction Set Computer), RISC (Reduced Instruction Set Computer), and VLIW (Very Long Instruction Word).

[0156] The quantization scheme has been exemplified above; the inverse quantization of data will be introduced below.

[0157] Figure 7 illustrates an apparatus 700 for dequantizing target data according to an embodiment of the present disclosure, comprising: a control circuit 710 configured to parse a conversion instruction for data conversion and to dequantize the target data during the data conversion process; and a processing circuit 720 configured to: perform a first conversion on the target data to obtain data to be dequantized; perform a second conversion on a scaling factor used for dequantization to obtain a dequantization scaling factor, wherein the second conversion corresponds to the first conversion; and perform a dequantization operation on the data to be dequantized using the dequantization scaling factor to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

[0158] As mentioned above, dequantization refers to the process of converting low-precision numbers into data of generally higher precision. Low-precision data can include at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, and FP12. While these data are convenient for storage and data processing, their precision is relatively low. High-precision data can include at least one of FP16, BF16, FP32, FP64, and TF32.

[0159] It is important to understand that the above text only provides some examples of target data (low-precision data) and inverse quantized data (high-precision data), and is not an exhaustive list of these data types.

[0160] Application number CN2020110363257 discloses "a data processing device, integrated circuit chip, equipment, and a method for implementing the same," which can implement various operations on multi-dimensional data (such as two-dimensional matrices, three-dimensional arrays, or higher-dimensional data) based on data conversion instructions. These operations include one or more of the following: multi-dimensional data bypass operations, multi-angle rotation operations, mirroring operations, or sequential transformation operations. The data conversion instructions can be, for example, any instruction involving multi-dimensional data conversion, such as the tilting instruction.

[0161] In this disclosure, data conversion and dequantization functions can be combined in a single data conversion instruction, thereby enabling data conversion and dequantization to be achieved with a single instruction.

[0162] The scaling factor used for dequantization can be the same as the scaling factor used during quantization. Figure 8 shows a flowchart of determining the scaling factor according to one embodiment of this disclosure.

[0163] As shown in Figure 8, according to one embodiment of this disclosure, determining the scaling factor may include: in operation S810, determining the maximum absolute value in a set of data to be quantized; in operation S820, obtaining the maximum value that the target data type after quantization of the data to be quantized can represent; in operation S830, determining the scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and in operation S840, converting the scaling factor from a first data type to a second data type to obtain the scaling factor, wherein the bit width of the second data type is smaller than the bit width of the first data type.

[0164] In some embodiments, the scaling factor can be further compressed to save storage space and transmission bandwidth.

[0165] According to one embodiment of this disclosure, the scaling factor can also be determined by: determining K elements P that are less than or equal to K elements. iThe maximum absolute value of the element is raised to the greatest square; this greatest square is divided by the element P. i The scaling factor is the maximum power of the value that the data type can represent.

[0166] The specific process of determining the scaling factor has been described in more detail in the "Exemplary MX Quantization Scheme" and "Exemplary Extended MX Quantization Method" above, and will not be repeated here.

[0167] According to one embodiment of the present disclosure, the processing circuit 720 is further configured to: quantize the set of data to be quantized using the scaling factor to obtain quantized data having the target data type.

[0168] This embodiment has been described above in conjunction with Figures 4b to 6, and will not be repeated here.

[0169] According to one embodiment of this disclosure, the first data type is the same as the original data type of the data to be quantized, or the bit width of the first data type is greater than the bit width of the original data type; and the second data type is represented as ExMy, x+y<bit width of the original data type, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, and y represents the mantissa bit width.

[0170] This implementation has already been described above in conjunction with the exemplary MX quantization scheme and the exemplary extended MX quantization scheme, and will not be repeated here.

[0171] Scaling factors can be cached in on-chip caches or cache components, so that when scaling factors are needed for calculations, these factors can be retrieved from on-chip caches or cache components.

[0172] In some embodiments, the scaling factor can adopt the aforementioned compressed format to save storage space and transmission bandwidth. In this case, the conversion instruction can be supplemented with corresponding indication information to indicate the compressed format of the scaling factor. At this time, the processing circuit 720 can be further configured to: before performing a second conversion on the scaling factor, in response to the conversion instruction indicating that the scaling factor is in a compressed format, decompress the compressed scaling factor to obtain decompressed scaling factors for the second conversion. In some implementations, the compressed format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. At this time, the processing circuit 720 can be further configured to: combine the shared exponent with the Z new mantissas respectively to obtain Z decompressed scaling factors for the second conversion.

[0173] Furthermore, since the target data has been transformed, meaning the position of each data point has changed, the scaling factor needs to be transformed accordingly.

[0174] Figures 9a and 9b illustrate an example of corresponding conversion of target data and scaling factor according to one embodiment of the present disclosure.

[0175] As shown in Figure 9a, assuming the target data matrix M is 64*64 in size, then this matrix M can include four target data submatrices M0-M3 of size 32*32. Assuming each scaling factor is used for a specific number of data (e.g., 32 data), the scaling factor is a 64*2 matrix C, which can contain four scaling factor submatrices C0-C3 of size 32*1. Among them, scaling factor submatrix C0 is used for target data submatrix M0, scaling factor submatrix C1 is used for target data submatrix M1, scaling factor submatrix C2 is used for target data submatrix M2, and scaling factor submatrix C3 is used for target data submatrix M3.

[0176] Suppose that matrix M is mirrored in the left and right directions, for example, the target data submatrices M0 and M1 are swapped, and the target data submatrices M3 and M2 are swapped, thus forming the data matrix M' to be dequantized; correspondingly, the scaling factor matrix also needs to be transformed in the same way, that is, the scaling factor submatrices C0 and C1 are swapped, and the scaling factor submatrices C3 and C2 are swapped, thus forming the dequantization scaling factor C', as shown in Figure 9b.

[0177] It is important to understand that Figures 9a and 9b only illustrate one example of matrix transformation. Multidimensional data and corresponding scaling factors can undergo any other necessary transformations, which will not be exhaustively listed here. It is also important to understand that while the transformation methods for target data and scaling factors described in this paper are generally the same, it is possible to make the transformed target data correspond to the scaling factors in other ways. For example, the pointers to the scaling factors can be transformed accordingly without transforming the data of the scaling factors themselves.

[0178] According to one embodiment of this disclosure, the first conversion operation and the dequantization operation are pipelined parallel operations.

[0179] Figure 10 shows a schematic diagram of a pipelined parallel operation according to one embodiment of the present disclosure.

[0180] As shown in Figure 10, in cycle 1, the first conversion operation is performed on the target data 0. At this time, no data has been received in the dequantization operation, so the device can be idle. Next, in cycle 2, the target data 0 after the first conversion operation moves to the dequantization operation point, so the dequantization operation can be performed on the target data 0 after the first conversion operation. At the same time, the first conversion operation can be performed on the next target data 1 after the target data 0. Then, in cycle 3, the target data 1 after the first conversion operation moves to the dequantization operation point, so the dequantization operation can be performed on the target data 1 after the first conversion operation. At the same time, the first conversion operation can be performed on the next target data 2 after the target data 1.

[0181] As shown in Figure 10, the first conversion operation and the dequantization operation are upstream and downstream of a pipelined operation. As the target data stream continues to enter, the upstream and downstream operations perform conversion and dequantization operations on different target data in parallel.

[0182] In the prior art, conversion operations and dequantization operations usually require different instructions and are performed separately. However, in the embodiments of this disclosure, conversion operations and dequantization operations can be performed in a streamlined manner using a single instruction, thereby improving operating efficiency.

[0183] According to one embodiment of this disclosure, receiving target data and acquiring scaling factors are performed alternately. For example, target data and scaling factors can be sent alternately via related instructions. This alternating sending of target data and scaling factors makes it easier to maintain synchronization between the target data and scaling factors.

[0184] According to one embodiment of this disclosure, the caching component may include two ping-pong operated registers to enable real-time access to the scaling factor. The basic principle of ping-pong operation is that it uses two parallel data registers, one for reading data and the other for writing data. When one register is full, the system immediately switches to the other register for read / write operations. In this way, the data processing module can continuously read data from one register while simultaneously writing new data to another, thereby achieving real-time data processing.

[0185] Figure 11 illustrates an apparatus for inverse quantization of target data according to an embodiment of the present disclosure, comprising: a first conversion component 1110, a second conversion component 1120, and an inverse quantization component 1130, wherein the first conversion component 1110 is used to perform a first conversion on the data to obtain data to be inverse quantized; the second conversion component 1120 is used to perform a second conversion on a scaling factor used for inverse quantization to obtain an inverse quantization scaling factor, wherein the second conversion corresponds to the first conversion; and the inverse quantization component 1130 is connected to the first conversion component 1110 and the second conversion component 1120 respectively, and is used to perform an inverse quantization operation on the data to be inverse quantized using the inverse quantization scaling factor to obtain inverse quantized data, wherein the bit width of the inverse quantized data is greater than the bit width of the target data.

[0186] As shown in Figure 11, the first transformation component 1110 receives the target data (e.g., multi-dimensional data) and transforms it. As described above, these transformations include one or more of bypass operations, multi-angle rotation operations, mirror operations, and sequential transformation operations.

[0187] After receiving the scaling factor, the second conversion component 1120 also performs a corresponding conversion on the scaling factor. The converted data to be dequantized and the dequantization scaling factor enter the dequantization component, and then the dequantization component 1130 performs calculations on the dequantized data and the dequantization scaling factor to obtain the final dequantized data.

[0188] The second conversion component 1120 may receive scaling factors from off-chip memory, but according to one embodiment of the present disclosure, the apparatus for dequantizing the target data may further include a cache component 1140 (as shown in FIG11) for caching the scaling factors on-chip, the cache component 1140 being connected to the second conversion component 1120 for obtaining the scaling factors from the on-chip cache component.

[0189] As shown in Figure 11, the first conversion component 1110 and the dequantization component are located upstream and downstream of the data pipeline, respectively. Therefore, the first conversion operation and dequantization operation can be performed on the target data in a pipelined parallel operation manner. The pipelined parallel operation has been described in conjunction with Figure 10 and will not be repeated here.

[0190] According to one embodiment of this disclosure, the first conversion component 1110 receives data and obtains scaling factors from the cache component 1140 alternately.

[0191] According to one embodiment of this disclosure, the cache component 1140 includes two ping-pong operated registers to enable real-time access to the scaling factor.

[0192] According to one embodiment of this disclosure, each scaling factor is used for a specific number of data to be inversely quantized.

[0193] According to one embodiment of this disclosure, a decompression circuit 1150 is further included, configured to: decompress the scaling factors of the compressed format before performing the second conversion on the scaling factors, to obtain decompressed scaling factors for provision to the second conversion component 1120. In some implementations, the compressed format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. In this case, the decompression circuit 1150 is further configured to: combine the shared exponent with the Z new mantissas respectively to obtain Z decompressed scaling factors.

[0194] According to one embodiment of this disclosure, the scaling factor is determined by: determining the maximum absolute value in a set of data to be quantized; obtaining the maximum value that the target data type after quantization of the data to be quantized can represent; determining the scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and converting the scaling factor from a first data type to a second data type to obtain the scaling factor, wherein the bit width of the second data type is smaller than the bit width of the first data type.

[0195] According to one embodiment of this disclosure, the scaling factors can be further compressed into a compressed format as follows: determining a shared exponent of Z scaling factors to be compressed, where Z ≥ 2; determining Z new mantissas corresponding to the Z scaling factors; and bundling the shared exponent and the Z new mantissas for output for association. In some implementations, determining the shared exponent may include setting the largest exponent among the Z scaling factors as the shared exponent. In some implementations, determining the Z new mantissas may include setting the new mantissas of scaling factors with exponents less than the largest exponent to 0; and setting the new mantissas of scaling factors with exponents equal to the largest exponent as the original mantissas. In some implementations, the second data type of the Z scaling factors to be compressed is the same. In some implementations, the Z scaling factors belong to Z adjacent groups of data to be quantized. In some implementations, Z is 2, 4, or 8.

[0196] According to one embodiment of this disclosure, the scaling factor can also be determined by: determining K elements P that are less than or equal to K elements. i The maximum absolute value of the element is raised to the greatest square; this greatest square is divided by the element P. i The scaling factor is the maximum power of the value that the data type can represent.

[0197] According to one embodiment of this disclosure, the first data type is the same as the original data type of the data to be quantized, or the bit width of the first data type is greater than the bit width of the original data type; and the second data type is represented as ExMy, x+y<bit width of the original data type, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, and y represents the mantissa bit width.

[0198] According to one embodiment of this disclosure, the target data type includes at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, and FP12; and the original data type of the dequantized data includes at least one of FP16, BF16, FP32, FP64, and TF32.

[0199] Figure 12 shows a flowchart of a method for dequantizing data, including: in operation S1210, performing a first transformation on the target data to obtain data to be dequantized; in operation S1220, performing a second transformation on the scaling factor used for dequantization to obtain a dequantization scaling factor, wherein the second transformation corresponds to the first transformation; and in operation S1230, performing a dequantization operation on the data to be dequantized using the dequantization scaling factor to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

[0200] According to another aspect of this disclosure, an integrated circuit device is also provided, including the device described above, such as the device shown in FIG7 or FIG11.

[0201] According to another aspect of this disclosure, a board includes an integrated circuit device as described above.

[0202] This disclosure also provides a computing device for inverse quantization, comprising: at least one processor; and at least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform the method described in FIG12.

[0203] This disclosure also provides a computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the method described in FIG12.

[0204] Figure 13 shows a block diagram of an apparatus for performing data conversion according to another embodiment of the present disclosure.

[0205] As shown in Figure 13, the apparatus for inverse quantization of target data disclosed herein also includes a data processing device 5100, which can, for example, convert FP6 type data into FP8 type data. It should be understood that this description uses FP6 and FP8 type data merely as examples and is not intended to limit the scope of protection of this disclosure.

[0206] Figure 14 shows a schematic diagram of the structure of a data processing apparatus 5100 according to one embodiment of the present disclosure.

[0207] The data processing device 5100 may include: a first control circuit 5110 configured to parse a first memory access instruction, the first memory access instruction being used for data transfer, and to perform data type conversion on the data during the data transfer process; a first processing circuit 5120 configured to: extract first data from a first source address according to the first memory access instruction; sparsify and convert the first data into second data, wherein the first data is densely arranged, and the first bit width w1 of the first data is lower than the second bit width w2 of the second data; and store the second data at a first destination address. The second data mentioned herein may be the "target data" as described herein.

[0208] According to one embodiment of this disclosure, the first source address may be located in off-chip memory 5200, and the first destination address may be located in on-chip memory 5300. In fact, the on-chip memory and off-chip memory described above are merely examples and not limitations on these memory addresses. Essentially, in the memory hierarchy, the farther the memory is from the processor, the slower its speed, but the larger its capacity. This disclosure does not limit the specific locations of the first source address and the first destination address; in the memory hierarchy, the first source address is farther from the arithmetic circuitry than the first destination address.

[0209] It should be understood that the data processing device 5100 and the on-chip memory 5300 may reside in the same chip 500, or the data processing device 5100 may exist independently. This disclosure does not impose any restrictions on the location of the data processing device 5100.

[0210] When the data processing device 5100 performs operations on the chip 500, it can retrieve data from the first source address of the off-chip memory 5200 for transformation and operation, and store the result data in the on-chip memory 5300. The data type stored in the on-chip memory 5300 (e.g., SRAM in Figure 3) is usually related to the data types that the data processing device can handle, and is typically used to store conventional data types, such as FP8, FP16, FP32, etc. However, in the off-chip memory (e.g., DDR memory or DRAM as shown in Figure 3), to fully utilize the storage space, it can store data of lower precision, such as data format obtained by quantization using the aforementioned particle quantization method (e.g., FP6 type data). Therefore, when reading FP6 type data from the off-chip memory 5200 and sending it to the on-chip memory 5300 for operation, the FP6 type data needs to be converted to FP8 type data suitable for the data processing device's operation. It should be understood that the data types described here are merely examples for ease of understanding; data types in this art refer to any available data types, such as FP type, Int type, TF type, etc.

[0211] It is understood that when moving data between on-chip memory 5300 and off-chip memory 5200, data type conversion can be performed. For example, the data moving and conversion operations can be implemented using a single data access instruction (IO instruction). In this disclosure, the instruction that moves and converts data (e.g., FP6 type data) from off-chip memory 5200 is referred to as the "first memory access instruction," while the instruction that moves and converts data (e.g., FP8 type data) from on-chip memory 5300 is referred to as the "second memory access instruction." It should be understood that these names are for ease of understanding only, and the first and second memory access instructions can be integrated into a single instruction, rather than necessarily being separate or independent instructions.

[0212] The following section will first use FP6 and FP8 as examples to introduce the implementation methods for data type conversion.

[0213] Figure 15 shows a flowchart of a method for converting low-bit-width data to high-bit-width data; Figures 16a to 16c exemplarily illustrate the data transformation when converting FP6 to FP8. It should be noted that in Figures 16a to 16c, the data bits on the right are lower than the data bits on the left, the diagonal lines represent actual data, and the blank areas represent blank spaces or invalid data.

[0214] As shown in Figure 15, this disclosure provides a method for converting first data having a first data type into second data having a second data type, wherein the first data is densely arranged, and the first bit width w1 of the first data type is lower than the second bit width w2 of the second data type. The method includes: in operation S1520, shifting the first data so that the starting bit of the first data is aligned with the nearest reference bit in the storage space; in operation S1530, splitting the first data according to the bit width ratio w1 / w2 to form multiple first intermediate data, the size of each first intermediate data not exceeding w1 / w2 times the maximum access capacity in each access cycle; in operation S1540, sparsifying the first intermediate data in the corresponding access cycle according to the bit width ratio to form multiple second intermediate data; and in operation S1550, converting the second intermediate data into second data having a second data type.

[0215] It should be noted that the technical solution disclosed herein can be applied to any type of data. For example, the first data type can be FP2, FP4, or FP6, while the second data type can be FP4, FP8, FP16, FP32, or FP64. It is important to note that the bit width of the first data must be lower than the bit width of the second data. Therefore, when the first data is of type PF2, the second data must be at least of type FP4; when the first data is of type PF4, the second data must be at least of type FP8; and when the data is of type PF6, the second data must be at least of type FP8, but can also be of type FP16, FP32, or FP64.

[0216] For example, when the first data type is FP6 and the second data type is FP16, there are two ways to convert them: The first way is to first convert the FP6 data to FP8, and then convert the FP8 data to FP16; the second way is to directly convert the FP6 data to FP16. However, it should be noted that the advantage of the first method is that after converting the FP6 data to FP8, calculations can be performed on the FP8 data, and then the result can be converted to FP16, which will improve computational efficiency.

[0217] The technical solution disclosed herein is not only applicable to FP type data, but also suitable for other types. According to another embodiment of this disclosure, the first data type can be int2, int4, or int6, and the second data type can be in8, int16, int32, or int64, etc.

[0218] Dense arrangement means that there are no blank or invalid data bits between adjacent data, thus maximizing the use of storage space. As shown in Figure 16a, the data starts from the start bit, occupies a part of storage cell 1, spans storage cells 2 to 7, and occupies a part of storage cell 8, with no blank or invalid data in between.

[0219] It should be explained that this disclosure illustrates data across three access cycles. In access cycle 0, the contents of storage units 0-3 are read; in access cycle 1, the contents of storage units 4-7 are read; and in storage cycle 3, the contents of storage units 8-11 are read. It should be noted that storage units 0-11 are essentially contiguous storage space; they are simply divided into three rows for ease of understanding in this application. The "content" referred to here can be actual existing data or blank space.

[0220] It should also be explained that the size of a storage unit is typically a power of 2. For example, taking FP8 data as an example, each number has a bit width of 8 bits, and each storage unit can store 4 bytes, or 32 bits. In this disclosure, a storage unit size of 32 bits is used, and 128 bits of data can be accessed in each access cycle. It should be understood that this is merely an example given for convenience and is not a limitation on the specific size of the data and storage space.

[0221] In this disclosure, the boundary of each storage cell is referred to as the "reference bit," which facilitates the orderly storage of data. Returning to Figure 16a, the starting bit of the data is located in storage cell 1, but it is not aligned with the reference bit. Therefore, it is necessary to first shift the data so that the starting bit is aligned with the reference bit.

[0222] As shown in Figure 16b, according to one embodiment of the present disclosure, the first data conversion circuit is further configured to perform the following operation based on the decoded first memory access instruction: before splitting the first data to form a plurality of first intermediate data, shifting the first data so that the starting bit of the first data is aligned with the nearest reference bit in the storage space.

[0223] Specifically, the data can first be shifted to the right so that the starting bit of the data is aligned with the reference bit at its lower position. During this shifting process, all the data is shifted to the lower position. Therefore, a portion of the data originally in storage unit 8 is shifted to storage unit 7, while storage unit 8 stores only a small portion of the data.

[0224] It is important to understand that the shift operation of S1520 is not a necessary operation. For data that is already aligned with the reference bit, there is no need to perform a shift operation.

[0225] It's also important to understand that shifting can also be done towards a higher reference bit, but shifting towards a lower bit is more efficient in utilizing storage space, while shifting towards a higher bit may require an additional access cycle. For example, if the data in access cycle 2 is full, then shifting towards a higher bit will move a portion of the data in storage cell 11 to the storage cell in the next cycle.

[0226] As shown in Figure 16b, after shifting the data, the resulting first intermediate data can be split to facilitate further data transformation. For ease of understanding, in Figures 16b and 16c, the data stored in each storage unit is numbered based on the storage cell, i.e., numbered 0-7. It is important to understand that these numbers are not based on the data itself, but rather on the storage cell. In other words, due to the dense storage of FP6 type data, the content stored in the same storage cell does not necessarily belong to the same data, but may belong to two different FP6 data sets.

[0227] According to one embodiment of the present disclosure, in each access cycle, the first intermediate data that has been split off is stored sequentially starting from the lowest reference bit of the corresponding storage space.

[0228] Specifically, when splitting, the bit width ratio of the first and second data needs to be considered. For example, the bit width ratio of FP6 and PF8 is 6 / 8, or it can also be expressed as 96 / 128. During concatenation, since the data in access cycle 0 is exactly 6 / 8 (or 96 / 128) of the maximum storage capacity (128 bits), data 0-2 can be concatenated together; while for access cycle 1, since data 3-6 has exceeded 6 / 8 (or 96 / 128) of the maximum storage capacity (128 bits), data 3-5 needs to be concatenated together; and data 6 is concatenated with data 7 in the next access cycle 2 so that data 6 and 7 can be stored in storage cycle 3.

[0229] Understandably, each storage unit can store a maximum of 32 bits of data, and an FP6 data set (e.g., containing 4 bytes) is 24 bits in size. Therefore, each storage unit has 8 bits of space left to store a portion of the next FP6 data set, and so on. It's important to understand that when storing these FP6 data sets in each storage unit, they are stored sequentially starting from the least significant bit.

[0230] According to one embodiment of this disclosure, the size of the first intermediate data stored earlier is not less than the size of the first intermediate data stored later.

[0231] According to this embodiment, preferably, when splicing data sequentially, as much data as possible should be spliced ​​each time, which will help avoid the data spliced ​​later occupying more access cycles.

[0232] Based on the above description, it can be understood that in the view shown in Figure 16c, the total space of storage cells 1-3 is 96 bits, which actually stores four PF6 type data. In this case, these four PF6 type data can be further sparsified, that is, these four PF6 type data are distributed across four storage cells, and the blank spaces are filled with blank or invalid data (e.g., all 0 data), thereby forming multiple second intermediate data. Specifically, every 6 bits can be stored in the first 6 bits of the 8-bit storage space, and then the remaining 2 bits of storage space can be filled with 0 or invalid data.

[0233] Next, the second intermediate data is converted into PF8 type data. This process can be carried out using any existing or future conversion method, which will not be detailed here.

[0234] According to one embodiment of the present disclosure, the method further includes determining a bit width ratio w1 / w2.

[0235] It's important to understand that the above operations are not always necessary. For example, if the data types of the first and second data are known or fixed, no steps are needed. However, if the first and second data are unknown, their types must be determined first, followed by the bit width ratio. This operation makes data type conversion adaptive.

[0236] According to one embodiment of this disclosure, the data is particle-quantized or dequantized data. It should be understood that the data disclosed herein can be conventional data or data that has undergone quantization or dequantization, and this disclosure does not limit the source of the data.

[0237] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0238] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An apparatus for inverse quantizing target data, comprising: A control circuit is configured to parse conversion instructions, the conversion instructions being used for data conversion, and to inverse quantize the target data during the data conversion process; Processing circuitry, configured for: The target data is subjected to a first transformation to obtain the data to be inversely quantized; A second transformation is performed on the scaling factor used for inverse quantization to obtain the inverse quantization scaling factor, wherein the second transformation corresponds to the first transformation; as well as The dequantization operation is performed on the data to be dequantized using the dequantization scaling factor to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

2. The apparatus of claim 1, further comprising obtaining the scaling factor from an on-chip cache component.

3. The apparatus according to claim 1 or 2, wherein, The target data is multi-dimensional data, and the first transformation includes performing one or more of the following operations on the multi-dimensional data: bypass operation, multi-angle rotation operation, mirror operation, and sequence transformation operation.

4. The apparatus according to any one of claims 1-3, wherein, The first conversion operation and the dequantization operation are pipelining parallel operations.

5. The apparatus according to claim 2, wherein, Receiving target data and obtaining scaling factors are done alternately.

6. The apparatus according to claim 5, wherein, The cache component includes two ping-pong operated registers to enable real-time access to the scaling factor.

7. The apparatus according to any one of claims 1-6, wherein, Each scaling factor is used for a specific amount of data to be inverted.

8. The apparatus according to any one of claims 1-7, wherein the processing circuitry is further configured to: Before performing the second conversion on the scaling factor, in response to the conversion instruction indicating that the scaling factor is in a compressed format, the compressed scaling factor is decompressed to obtain the decompressed scaling factor for the second conversion.

9. The apparatus according to claim 8, wherein, The compression format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. The processing circuit is further configured to: The shared index is combined with the Z new tail numbers to obtain Z decompressed scaling factors for the second conversion.

10. The apparatus according to any one of claims 1-9, wherein, The scaling factor is determined as follows: Determine the maximum absolute value in a set of data to be quantified; Obtain the maximum value that the target data type can represent after the data to be quantized; Based on the ratio between the maximum absolute value and the maximum value, the scaling factor corresponding to the set of data to be quantized is determined; The scaling factor is converted from a first data type to a second data type to obtain a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type.

11. The apparatus according to claim 10, wherein, The target data is obtained as follows: the set of data to be quantized is quantized using the scaling factor to obtain quantized data with the target data type.

12. The apparatus according to claim 10 or 11, wherein The first data type is the same as the original data type of the data to be quantized, or the bit width of the first data type is greater than the bit width of the original data type; and The second data type is represented as ExMy, where x + y < the bit width of the primitive data type. E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width.

13. The apparatus according to any one of claims 10-12, wherein, The scaling factor is further compressed into a compressed format as follows: Determine the shared exponent of the Z scaling factors to be compressed, Z≥2; Determine the Z new mantissas corresponding to the Z scaling factors; as well as The shared index is bundled with the Z new tail numbers and output for association.

14. The apparatus of claim 13, wherein determining the sharing index comprises: Set the largest index among the Z scaling factors as the shared index.

15. The apparatus of claim 14, wherein determining the Z new tail numbers comprises: Set the new mantissa of the scaling factor whose exponent is less than the maximum exponent to 0; as well as Set the new mantissa, whose exponent is equal to the scaling factor of the maximum exponent, as the original mantissa.

16. The apparatus according to any one of claims 13-15, further comprising one or more of the following: The second data type of the Z scaling factors to be compressed is the same; The Z scaling factors belong to Z adjacent groups of data to be quantized; Z can be 2, 4, or 8.

17. The apparatus according to any one of claims 1-7, wherein, The scaling factor is determined as follows: Determine the number of elements P that are less than or equal to K. i The largest power of the absolute value in the range; Divide the largest power by element P i The scaling factor is the maximum power of the value that the data type can represent.

18. The apparatus according to any one of claims 1-17, wherein, The target data type includes at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, and FP12; and The original data type of the inverse quantization data includes at least one of FP16, BF16, FP32, FP64, and TF32.

19. An apparatus for inverse quantizing target data, comprising: The system comprises a first conversion component, a second conversion component, and an inverse quantization component, wherein... The first conversion component is used to perform a first conversion on the data to obtain the data to be inversely quantized; The second conversion component is used to perform a second conversion on the scaling factors used for inverse quantization to obtain inverse quantization scaling factors, wherein the second conversion corresponds to the first conversion; and The dequantization component is connected to the first conversion component and the second conversion component respectively, and is used to perform dequantization operation on the data to be dequantized using the dequantization scaling factor to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

20. The apparatus of claim 19, further comprising a caching component for caching the scaling factor on-chip, wherein the second conversion component is connected to the caching component for retrieving the scaling factor from the on-chip caching component.

21. The apparatus according to claim 20, wherein, The target data is multi-dimensional data, and the first transformation includes performing one or more of the following operations on the multi-dimensional data: bypass operation, multi-angle rotation operation, mirror operation, and sequence transformation operation.

22. The apparatus according to any one of claims 19-21, wherein, The first conversion operation performed at the first conversion component and the dequantization operation performed at the dequantization component are pipelining parallel operations.

23. The apparatus according to claim 20, wherein, The first conversion component receives data and the caching component obtains the scaling factor alternately.

24. The apparatus according to claim 23, wherein, The cache component includes two ping-pong operated registers to enable real-time access to the scaling factor.

25. The apparatus according to any one of claims 19-24, wherein, Each scaling factor is used for a specific amount of data to be inverted.

26. The apparatus according to any one of claims 19-25, further comprising a decompression circuit configured for: Before performing the second conversion on the scaling factor, the compressed scaling factor is decompressed to obtain the decompressed scaling factor, which is then provided to the second conversion component.

27. The apparatus according to claim 26, wherein, The compression format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. The decompression circuit is further configured to: The shared index is combined with the Z new mantissas to obtain Z decompressed scaling factors.

28. The apparatus according to any one of claims 19-27, wherein, The scaling factor is determined by identifying the maximum absolute value in a set of data to be quantized. Obtain the maximum value that the target data type can represent after the data to be quantized; Based on the ratio between the maximum absolute value and the maximum value, the scaling factor corresponding to the set of data to be quantized is determined; The scaling factor is converted from a first data type to a second data type to obtain a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type.

29. The apparatus according to claim 28, wherein, The first data type is the same as the original data type of the data to be quantized, or the bit width of the first data type is greater than the bit width of the original data type; and The second data type is represented as ExMy, where x + y < bit width of the original data type, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, and y represents the mantissa bit width.

30. The apparatus according to any one of claims 28-29, wherein, The scaling factor is further compressed into a compressed format as follows: Determine the shared exponent of the Z scaling factors to be compressed, Z≥2; Determine the Z new mantissas corresponding to the Z scaling factors; as well as The shared index is bundled with the Z new tail numbers and output for association.

31. The apparatus of claim 30, wherein determining the sharing index comprises: Set the largest index among the Z scaling factors as the shared index.

32. The apparatus of claim 31, wherein determining the Z new tail numbers comprises: Set the new mantissa of the scaling factor whose exponent is less than the maximum exponent to 0; as well as Set the new mantissa, whose exponent is equal to the scaling factor of the maximum exponent, as the original mantissa.

33. The apparatus according to any one of claims 30-32, further comprising one or more of the following: The second data type of the Z scaling factors to be compressed is the same; The Z scaling factors belong to Z adjacent groups of data to be quantized; Z can be 2, 4, or 8.

34. The apparatus according to any one of claims 19-27, wherein, The scaling factor is determined as follows: Determine the number of elements P that are less than or equal to K. i The largest power of the absolute value in the range; Divide the largest power by element P i The scaling factor is the maximum power of the value that the data type can represent.

35. The apparatus according to any one of claims 19-34, wherein, The target data type includes at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, and FP12; and The original data type of the inverse quantization data includes at least one of FP16, BF16, FP32, FP64, and TF32.

36. A method for inverse quantization of data, comprising: The target data undergoes a first transformation to obtain the data to be inversely quantized. A second transformation is performed on the scaling factor used for inverse quantization to obtain the inverse quantization scaling factor, wherein the second transformation corresponds to the first transformation; as well as The dequantization operation is performed on the data to be dequantized using the dequantization scaling factor to obtain dequantized data, wherein the bit width of the dequantized data is greater than the bit width of the target data.

37. An integrated circuit device comprising the device according to any one of claims 1 to 35.

38. A board comprising the integrated circuit device according to claim 37.

39. A computing device for inverse quantization, comprising: At least one processor; as well as At least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform the method of claim 36.

40. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the method according to claim 36.

Images