Data processing apparatus, processor, board card, and data processing method

Abstract

A data processing apparatus, a processor, a board card, and a data processing method. The data processing apparatus may be comprised by a combined processing apparatus (20), the combined processing apparatus (20) comprising a computing apparatus (201), an interface apparatus (202), a processing apparatus (203) and a storage apparatus (204). The computing apparatus (201) is configured to execute an operation specified by a user, so as to execute deep learning or machine learning calculation. The computing apparatus (201) may interact with the processing apparatus (203) by means of the interface apparatus (202), so as to jointly complete the operation specified by the user. The interface apparatus (202) is used to transmit data and a control instruction between the computing apparatus (201) and the processing apparatus (203). The storage apparatus (204) is used to store data of the computing apparatus (201) and the processing apparatus (203). The data processing apparatus provides a vector computation scheme integrating different types of fine-grained quantization, so that processing can be simplified, and the advantages of a low bit width operation and small memory occupation of a fine-grained quantization format are fully utilized.

Description

Data processing device, processor, circuit board and data processing method Cross-reference to related applications

[0001] This application claims priority to Chinese patent applications filed on June 21, 2024, with application number 202410813886.5, entitled "Data Processing Apparatus, Processor, Board and Data Processing Method" and filed on September 25, 2024, with application number 202411347356.2, entitled "Data Processing Apparatus, Processor, Board and Data Processing Method". Technical Field

[0002] This disclosure generally relates to the field of computer processors. More specifically, this disclosure relates to a data processing apparatus, a processor, a circuit board, and a data processing method implemented by the data processing apparatus. Background Technology

[0003] With the rapid development of artificial intelligence, the computational load and number of parameters in neural network models are continuously increasing. Research shows that during training and inference, reducing the precision of floating-point operations can effectively improve operating speed, reduce power consumption, and decrease chip area.

[0004] To address this, AI vendors have designed various floating-point formats, such as BF16 and TF32, to achieve a balance between AI system performance and accuracy. These floating-point formats each have their own application scenarios and advantages; however, the diverse or specialized nature of these formats also presents migration challenges. For example, a model trained by vendor A using FP6 cannot run on vendor B's inference hardware.

[0005] To establish a unified industry standard, the Open Compute Project (OCP) introduced the MX (MicroXcaling) quantization format. The MX quantization format is an interoperable quantization data format with shared, fine-grained block sizes. MX-compliant data formats support AI training and inference, featuring lower bit-width arithmetic operations and smaller memory footprints, thus driving improvements in hardware performance and efficiency, thereby reducing overhead and operating costs.

[0006] In view of this, given that vector operations account for a large proportion of neural network operations, it is necessary to develop an operation mode and corresponding hardware that can perform MX quantization / dequantization processing during vector operations, so as to fully leverage the advantages of the MX quantization format. Summary of the Invention

[0007] To address at least one or more of the technical problems mentioned above, this disclosure proposes a scheme for integrating vector MX quantization / dequantization during vector operations in several aspects. Furthermore, an improved MX quantization scheme is also provided.

[0008] In a first aspect, this disclosure provides a data processing apparatus, comprising: a control circuit configured to parse vector operation instructions, the vector operation instructions indicating to perform particle quantization or dequantization on an input vector and then perform an operation, or to perform an operation on an input vector and then perform particle quantization or dequantization on the operation result; and an operation circuit configured to perform particle quantization or dequantization on the input vector and then perform an operation, or to perform an operation on the input vector and then perform particle quantization or dequantization on the operation result, according to the vector operation instructions.

[0009] In a second aspect, this disclosure provides a processor that includes the data processing apparatus described in the first aspect.

[0010] In a third aspect, this disclosure provides a board that includes the processor described in the second aspect above.

[0011] In a fourth aspect, this disclosure provides a data processing method implemented by a data processing apparatus, the data processing apparatus including a control circuit and a computation circuit, the method comprising: the control circuit parsing vector computation instructions, the vector computation instructions being used to perform particle quantization or dequantization on an input vector and then perform computation, or to perform computation on an input vector and then perform particle quantization or dequantization on the computation result; and the computation circuit performing particle quantization or dequantization on the input vector and then perform computation, or to perform computation on the input vector and then perform particle quantization or dequantization on the computation result, according to the parsed vector computation instructions.

[0012] In the vector operation scheme provided in this disclosure embodiment, the quantization / dequantization process of data can be integrated into the vector operation, eliminating the need for separate quantization / dequantization of the data. This avoids separate quantization / dequantization processing and fully utilizes the advantages of low bit width and small memory footprint of the particle quantization format in vector operations. Here, the particle quantization format may include the MX quantization format proposed by OCP and the extended MX quantization format proposed in this disclosure embodiment. Optionally or additionally, in some embodiments, the scaling factor obtained by extended MX quantization can also be compressed, thereby reducing the waste of storage space and bandwidth. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0014] Figure 1 shows a structural diagram of the board card according to an embodiment of this disclosure;

[0015] Figure 2 shows a structural diagram of the combined processing apparatus according to an embodiment of this disclosure;

[0016] Figure 3 shows a schematic diagram of the internal structure of the processing core when the computing device is a single-core or multi-core device;

[0017] Figure 4 shows a schematic diagram of the MX quantization data format;

[0018] Figure 5 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure;

[0019] Figure 6 shows a schematic block diagram of the comparison circuit of some embodiments of this disclosure;

[0020] Figure 7 illustrates a schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure;

[0021] Figure 8 shows an exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of this disclosure. Detailed Implementation

[0022] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0024] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0025] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0026] First, we provide explanations of the technical terms that may be used in this disclosure.

[0027] Floating-point numbers: The IEEE floating-point standard uses V = (-1). sign *mantissa*2 E A number is represented in the form of . Here, `sign` is the sign bit, where 0 represents a positive number and 1 represents a negative number; `E` represents the exponent (or index), which is a weighted sum of 2 raised to the power of `E` (which may be a negative power); `mantissa` represents the mantissa, a binary fraction. Floating-point numbers are represented in computers using three fields, each encoded separately:

[0028] (1) A single sign bit directly encodes the symbol s;

[0029] (2) The exponent field of the k-bit is encoded as exp=e(k-1)......e(1)e(0);

[0030] (3) The n-digit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent field is all 0.

[0031] Fixed-point numbers are a way of representing numbers in computers, characterized by a fixed decimal point. Fixed-point numbers can represent integers or decimals, depending on the implicit position of the decimal point. A fixed-point number consists of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision.

[0032] Bit width: How many bits are used to represent data.

[0033] Quantization: The process of converting high-precision numbers into lower-precision data that takes up less memory.

[0034] Dequantization: The process of converting low-precision numbers into higher-precision data.

[0035] FP*: Represents a floating-point data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0036] BF16 and TF32 are data types suitable for deep learning, which improve computational performance by sacrificing precision and range.

[0037] int*: Represents an integer data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0038] ExMy: A data format where E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width.

[0039] MX: refers to the Microscaling specification (see “OCP Microscaling Formats (MX) Specification Version 1.0”).

[0040] Extended MX: refers to a quantization scheme proposed in the embodiments disclosed herein.

[0041] Granular quantization: This includes MX and extended MX, as well as other quantizations with similar data structures. The quantization granularity is determined based on the amount of data that the arithmetic unit can process at one time.

[0042] It should be noted that in this disclosed embodiment, the data to be quantized or dequantized can be various data in the neural network, including but not limited to input neurons, weights, output neurons, gradients, etc. The neural network provided in this disclosed embodiment can be applied to various fields and can process various types of data. For example, in image recognition and processing, facial recognition, object detection, image classification, and medical image analysis can include data types such as pixel data and image features; in natural language processing (NLP), language translation, sentiment analysis, text summarization, and speech recognition can include data types such as text data, word embeddings, and sentence structure; in speech recognition and processing, intelligent assistants, automatic caption generation, and speech-to-text conversion can include data types such as audio signals and spectrograms; in recommendation systems, personalized content recommendation, product recommendation, and advertising can include data types such as user behavior data, item features, and rating data; in healthcare, disease diagnosis, drug discovery, and gene sequence analysis can include data types such as medical records, biomarker data, and genomic data; in the financial field, risk assessment, fraud detection, and stock market prediction can include data types such as transaction data, user credit scores, and market data; and in autonomous driving, vehicle perception, decision-making, and path planning can include data types such as sensor data, environmental features, and traffic signals. In gaming and entertainment, AI applications such as game AI, virtual reality, and animation generation can include data types such as game state data and user interaction data. In scientific research, applications such as physics simulation, chemical compound prediction, and astrophysical analysis can include data types such as experimental data, simulation results, and observational data. Generative AI applications, on the other hand, utilize complex algorithms, models, and rules to learn from large-scale datasets to create new, original content. This includes, but is not limited to, creating various types of content such as text, images, audio, video, and code. Correspondingly, the input neuron data processed can be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc., and the corresponding output neuron data can also be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc. The output of the neural network can include a probability score of an image belonging to a specific object category, a probability score of a document relating to a specific topic, a probability score of the correct translation of a text fragment in the target language from a text fragment in the source language, or a probability score of the correct transcription of a text fragment from spoken language, etc. These output data can also be quantized or unquantized. By quantizing / dequantizing the data, the processing efficiency of these tasks can be maintained while reducing the computational load and saving computing resources.The data quantization scheme of this disclosure embodiment can also be used in the inference and training of neural networks. By performing the quantization process described in this disclosure embodiment on the data of the neural network, the accuracy of the task can be guaranteed when the neural network performs one or more of the aforementioned processing tasks, while improving the speed and performance of the neural network model's inference and training.

[0043] Exemplary hardware architecture

[0044] Figure 1 shows a schematic diagram of the structure of a board 10 according to an embodiment of this disclosure. As shown in Figure 1, the board 10 includes one or more chips 101, which are system-on-chips (SoCs) that integrate one or more combined processing devices. The combined processing device is an artificial intelligence computing unit used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant characteristic of cloud intelligence applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. The board 10 of this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0045] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as PCIe (Peripheral Component Interconnect express) interface.

[0046] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller, also known as a microcontroller unit (MCU).

[0047] Figure 2 is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. As shown in Figure 2, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0048] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0049] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0050] Processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0051] Storage device 204 is used to store data to be processed. It may be DRAM (Dynamic Random Access Memory), DDR (Double Data Rate) memory, typically 16G or larger, and is used to store data of computing device 201 and / or processing device 203.

[0052] Figure 3 shows a schematic diagram of the internal structure of the processing core when the computing device 201 in Figure 2 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31 (also called a controller), an arithmetic module 32 (also called an arithmetic unit), and a storage module 33 (also called a memory).

[0053] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0054] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0055] Storage module 33 is used to store or move relevant data, including Neuron RAM (NRAM) 331, Weight RAM (WRAM) 332, and Direct Memory Access (DMA) 333. NRAM 331 stores input neurons, output neurons, and intermediate results; WRAM 332 stores the convolution kernels, i.e., weights, of the deep learning network; DMA 333 is connected to DRAM 204 via bus 34 and is responsible for data transfer between computing device 301 and DRAM 204. It should be noted that NRAM and WRAM here can be two storage regions formed by dividing the same memory in logical storage space, or they can be two independent memories; no specific limitation is made here.

[0056] Exemplary MX quantization data

[0057] Figure 4 illustrates the MX quantized data format, specifically its components. As shown, the MX quantized data format is characterized by three parts: the scaling factor (X, shared scale), the scaling block size (K, scaling block size), and the private element data within the block (P). i ).

[0058] The scaling factor X relates to the data type or encoding method of the scaling factor. Private element (P) i It also involves its data type or encoding method. The scaling block size (K) represents the private element P within the block. i The number of elements P within the block. i All elements have the same data type and therefore the same bit width. The scaling factor X is shared among these K elements. The data type of the elements and the data type of the scaling factor can be chosen independently. In a sense, MX can be viewed as a mechanism for constructing vector data types based on scalar data types. The values ​​V1,…,V represented in an MX block are... k It can be determined as follows: V i =XP i .

[0059] As shown in the figure, W represents the number of bits used to encode the shared scaling factor X, that is, the bit width of the scaling factor X; d represents the number of bits used to represent each element P. i The number of bits, i.e., P i The bit width is . Therefore, for an MX block containing K elements, it can be encoded using (W+Kd) bits.

[0060] Table 1 shows some common MX formats and related parameters. Table 1

[0061] In Table 1 above, ExMy represents a scalar format notation with x exponent bits and y mantissa bits. For example, FP8 (E4M3) represents an FP8 format with 1 sign bit, 4 indicator bits, and 3 mantissa bits. When y is 0 (e.g., E8M0), the format does not include a sign bit.

[0062] Furthermore, in MX quantization, the scaling factor X is calculated as follows:

[0063] (1) Determine the number of elements P that are less than or equal to K. i The largest power of the absolute value in the range;

[0064] (2) Divide the largest power of 2 by element P. i The data type can represent the maximum power of 2, and the calculated result is set as the scaling factor Scale X.

[0065] The inventors have discovered that, for MX quantization, processing the original data into ultra-low precision data types such as F4 (representing a 4-bit data format) leads to a sharp decrease in the accuracy of large models, significantly impacting the accuracy of the computational results. Taking image recognition tasks as an example, task accuracy or processing performance is reflected in the accuracy of the image recognition results, i.e., whether the image recognition computation is accurate. The quantized image data obtained after quantization using the aforementioned quantization method affects the accuracy of the image recognition results.

[0066] The inventors also discovered that if the data type of the scaling factor used in the quantization method is the same as the original data type, although the above-mentioned problem of decreased accuracy can be solved to some extent, this method will lead to greater resource overhead (such as storage space, area overhead required for hardware support, etc.).

[0067] Exemplary Extended MX Quantization Method

[0068] Based on the above findings, this disclosure provides a novel data quantization scheme. This scheme determines a scaling factor based on the maximum absolute value of the data to be quantized, and obtains a scaling coefficient by converting the scaling factor to a different data type. Since the converted scaling coefficient has a non-zero mantissa, the accuracy of the quantized data is improved, resulting in processing tasks performed on the quantized data according to the embodiments of this disclosure exhibiting better accuracy. Furthermore, setting the quantization granularity based on the hardware's single-processing capacity facilitates the quantization of a block of granularity in a single hardware operation. This makes online quantization / dequantization more convenient, facilitating pipelined design and enabling neural network models using this hardware as a computing platform to perform quantization operations at various stages, including inference and training, thereby improving the computational efficiency of the neural network model.

[0069] In the extended MX quantization scheme, the scaling factor is calculated as follows:

[0070] (1) Determine the maximum absolute value in a set of data to be quantified;

[0071] (2) Obtain the maximum value that the target data type can represent after the data to be quantized is quantized;

[0072] (3) Determine the scaling factor corresponding to this set of data to be quantized based on the ratio between the maximum absolute value and the maximum value;

[0073] (4) Convert the scaling factor from the first data type to the second data type to obtain the scaling coefficient, wherein the bit width of the second data type can be smaller than the bit width of the first data type.

[0074] The target data type is the data type of the quantized data obtained after quantizing the data to be quantized. Before quantization, the target data type can be set as needed, and its precision is generally lower than that of the original data type of the data to be quantized. In some embodiments, the original data type of the data to be quantized may include at least one of FP16, BF16, FP32, FP64, TF32, etc.; the target data type may include at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, etc. Among these, floating-point data types such as FP8, FP6, and FP4 can support multiple data formats. Taking FP4 as an example, it can be divided in several ways according to the rule of "sign bit + exponent bit + mantissa bit," with the sign bit typically occupying 1 bit. Common combinations of exponent and mantissa bits include E3M0 and E2M1. Similarly, FP8 can support data formats such as E4M3 and E5M2, and FP6 can support data formats such as E3M2 and E2M3.

[0075] Due to the bit width limitations of each data type, each data type has a maximum representable value. For example, the maximum representable value when the target data type is FP4 is 6; the maximum representable value when the target data type is int8 is 127. Typically, there is a one-to-one correspondence between the maximum value and the target data type, which can be obtained through methods such as table lookup.

[0076] Since the data type of the absolute maximum value is the same as the original data type of the data to be quantized, the data type of the scaling factor obtained by the ratio (i.e., the first data type) is the same as the original data type of the corresponding data to be quantized, or the bit width of the first data type can be greater than the bit width of the original data type.

[0077] The first data type is the scaling factor's data type, and the second data type is the scaling coefficient's data type. Data type conversion makes the bit width of the second data type smaller than that of the first data type, thus the precision of the scaling coefficient is less than the precision of the scaling factor. In some embodiments, the conversion from scaling factor to scaling coefficient can be achieved using a conversion instruction, such as `cvt_to_scale_dtype()`. In other embodiments, the conversion supports multiple round modes, such as `rn(round to nearest, ties to even)`.

[0078] In some other embodiments, the second data type can be represented as ExMy, where E represents the exponent bit, M represents the mantissa bit, x represents the bit width of the exponent bit, y represents the bit width of the mantissa bit, and x + y < the bit width of the original data type. For example, assuming the original data type is FP16, then x + y < 16. The specific values of x and y can be set according to requirements or the hardware environment, x is a positive integer, and y ≥ 0. In some other embodiments, 3 ≤ x + y < the bit width of the original data type. In some embodiments, 8 ≤ x + y < the bit width of the original data type. In some embodiments, the second data type can include at least one of, for example, E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, E3M9, etc. In some preferred embodiments, y > 0. In further preferred embodiments, the second data type can be E8M2 or E8M3.

[0079] For the sake of easy understanding, still taking the original data type as FP16 as an example, when 8 < x + y < the bit width of the original data type, the bit width (i.e., x + y) represented by the second data type ExMy of the scaling factor can be between 8 bits and 16 bits. Using such a scaling factor to perform quantization processing on the operation data of the neural network and using the quantized data obtained after quantization for inference and training of the neural network, etc., the accuracy of the neural network obtained is between the 8-bit representation (such as E8M0 represented in MX) and the original 16-bit representation (such as E5M10, E8M7, i.e., FP16, BF16). Compared with the MX format, such a scaling factor in the embodiments of the present disclosure is beneficial to improving the accuracy of the quantized data; compared with the case where the data type of the scaling factor is the same as the original data type, such a scaling factor in the embodiments of the present disclosure is beneficial to reducing hardware resource overhead or storage overhead, etc.

[0080] Furthermore, since the bit width of the mantissa bit affects the quantization accuracy, a larger bit width of the mantissa bit can provide a scaling factor with higher accuracy, thereby retaining more details of the original data during the quantization process. Therefore, compared with the scaling factor of the E8M0 format in MX quantization, when the bit width y of the mantissa bit of the scaling factor in the embodiments of the present disclosure is greater than 0, it is beneficial to improve the accuracy of the quantized data and the accuracy of the processing task.

[0081] Furthermore, this disclosed embodiment enables the scaling factor to support multiple data types by performing data type conversion, thereby allowing for more flexible adaptation to different data characteristics and precision requirements. Regarding the selection of the exponent and mantissa bit widths of the scaling factor, the distribution characteristics of the data to be quantized can be considered. A larger exponent bit width allows for the representation of more values. If the range of values ​​in a set of data to be quantized varies significantly, the exponent bit width of the scaling factor can be set larger; conversely, if the range of values ​​in a set of data to be quantized varies only slightly, the exponent bit width of the scaling factor can be set smaller, while the mantissa bit width can be set larger.

[0082] For example, suppose the second data type needs to be set to 8 bits. If the absolute value range of a set of data to be quantized involves 2... -127 ~2 127 Then the second data type can be set to E8M0; if the absolute value range of a set of data to be quantized involves 2 -4 ~2 4 Then the second data type can be set to E3M5. Of course, it is understandable that the bit width of the second data type is not limited to 8 bits, and can also be set to 5 bits, 4 bits, etc., as needed.

[0083] After determining the scaling factor in the above manner, the scaling factor can be used to quantize the corresponding data to be quantized, or to dequantize the quantized data.

[0084] In some embodiments, quantization processing may include: dividing the data to be quantized by a scaling factor to obtain an intermediate quantization result; and performing data type conversion on the intermediate quantization result to obtain quantized data with the target data type. This data type conversion can also be implemented using a conversion instruction, such as `cvt_to_scale_dtype()`. This conversion instruction can be a hardware or software instruction at various levels, such as processor microinstructions, ISA instructions, or function instructions implemented in high-level languages ​​at various software levels.

[0085] Dequantization is generally the process of restoring low-precision data to high-precision data. In some embodiments, dequantizing quantized data using a scaling factor may include multiplying the quantized data with the scaling factor. In other embodiments, the quantized data may be converted from a target data type to a desired data type before being multiplied with the scaling factor, wherein the precision of the desired data type is higher than that of the target data type. In still other embodiments, dequantization can restore the quantized data to its original state before the quantization operation.

[0086] As can be seen from the second data type of the scaling factor obtained from the previous conversion, the bit width of a single scaling factor is 3 to 15 bits. For example, E8M4 requires 12 bits, which typically requires 2 bytes (2B) of storage space, so it will definitely not be fully utilized. This inevitably leads to a waste of storage space and transmission bandwidth. To solve this problem, in some embodiments, two or more scaling factors can be compressed, thereby saving storage space and bandwidth.

[0087] In some embodiments, after quantizing two or more sets of data to be quantized using the aforementioned method to obtain two or more scaling factors, these two or more scaling factors can be compressed. Then, the compressed scaling factors are used to quantize the corresponding sets of data to be quantized, respectively, to output quantized data with the target data type.

[0088] Specifically, after obtaining the scaling factors as described above, a compression process can be included, which specifically includes: determining the shared exponent of the Z scaling factors to be compressed, where Z ≥ 2; determining the Z new mantissas corresponding to these Z scaling factors; and bundling the shared exponent with the Z new mantissas for output for association. That is, after compression, the exponents of the Z scaling factors are reused, and their mantissas are different, thereby saving at least Z-1 exponent bits.

[0089] In some implementations, determining the shared exponent for the Z scaling factors to be compressed may involve setting the largest exponent among these Z scaling factors as the shared exponent. Choosing the largest exponent as the shared exponent ensures the precision of larger numbers as much as possible, since larger numbers typically have a greater impact on the calculation results.

[0090] In some implementations, determining the Z new mantissas corresponding to the Z scaling factors can include: setting the new mantissa of scaling factors whose exponent is less than the maximum exponent to 0; and setting the new mantissa of scaling factors whose exponent is equal to the maximum exponent to the original mantissa. It can be understood that for scaling factors whose exponent is equal to the maximum exponent, since the exponent remains unchanged, its mantissa remains unchanged; however, for scaling factors whose exponent is less than the maximum exponent, after the exponent is scaled up to the maximum exponent, to maintain equivalence, the mantissa would be proportionally reduced, exceeding the range of values ​​that the mantissa can represent. Therefore, its new mantissa is set to 0 here. This setting will result in some precision loss, but compared to the storage and bandwidth gains, this precision loss is acceptable in some cases.

[0091] In some implementations, the shared exponent and the Z new mantissas can be bundled together in sequence for association. For example, first the shared exponent, then the new mantissa of the first scaling factor, the new mantissa of the second scaling factor, and so on up to the new mantissa of the Z scaling factor.

[0092] Next, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized, so as to output quantized data with the target data type. Since the exponent and mantissa of the scaling factors with an exponent smaller than the maximum exponent will change after compression, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized separately in order to maintain the consistency of quantization and dequantization.

[0093] In some implementations, the second data type of the Z scaling factors to be compressed is the same. For example, all Z scaling factors are of type E8M4. In this way, the shared exponent bit width after compression is 8 bits, and the Z new mantissas are 4Z bits.

[0094] In some implementations, the Z scaling factors belong to Z adjacent groups of data to be quantized. This allows the data to be processed sequentially without requiring additional information to indicate the scaling factors that are compressed together.

[0095] In some implementations, Z can be 2, 4, 8, or other values. Those skilled in the art can choose an appropriate Z value based on the number K of data to be quantized in a set of data. For example, when K is small, the value of Z can be larger; when K is large, the value of Z needs to be smaller, thereby ensuring that the benefit of using the compression scheme is greater than the benefit of amplifying K by Z times.

[0096] Similarly, when dequantizing extended MX quantized data using a compressed format, the scaling factors need to be decompressed first. Specifically, during dequantization, in response to the scaling factors being in a compressed format, the compressed scaling factors are decompressed; and the decompressed scaling factors are used to dequantize the quantized data.

[0097] Based on the compression method and bundling approach described above, the bundled Z scaling factors can be decoded into Z independent scaling factors. Specifically, the shared exponent can be combined with each of the Z new mantissas to obtain the Z decompressed scaling factors.

[0098] It is understood that in some embodiments, in addition to calculating and storing the scaling factor, the reciprocal of the scaling factor is also calculated and stored. This facilitates converting the division operation between the data to be quantized and the scaling factor into a multiplication operation between the data to be quantized and the reciprocal of the scaling factor during the quantization process. The reciprocal of the scaling factor has the same data type as the scaling factor. Therefore, the compression and decompression processes described above can also be applied to the reciprocal of the scaling factor, which will not be described in detail here.

[0099] In the extended MX quantization scheme described above, the scaling factor is determined based on the maximum absolute value of the data to be quantized. This ensures the accuracy of the data after quantization based on the maximum absolute value. Furthermore, since the larger the value of the data to be quantized during the quantization process, the greater the loss of accuracy and the greater the impact on the task calculation results, ensuring the accuracy of the data based on the maximum absolute value helps to reduce the loss of task accuracy. Therefore, compared to methods such as calculating the scaling factor in MX, the extended MX quantization scheme is better at ensuring task accuracy.

[0100] Furthermore, compared to the quantization method where the data type of the scaling factor is the same as the original data type, the scaling factor with a second data type obtained by the number of revolutions in this disclosed embodiment can reduce the storage space required to store the scaling factor and the hardware overhead required to execute the task while ensuring the accuracy of the task.

[0101] In some embodiments, the number of data to be quantized in a set of data to be quantized can be determined based on the single processing volume and / or the number of input ports of the processing circuit performing the quantization process; or the number of data to be quantized in a set of data to be quantized can be determined based on the single operation volume and / or the number of input ports of the arithmetic unit that performs the operation using the quantized data.

[0102] For example, the number of data points to be quantized in a set of data can be determined based on the processing capacity of the processing circuit (i.e., the amount of data that can be processed in a single run), or based on the processing capacity of the arithmetic unit that performs operations such as multiply-accumulate or vector inner product using quantized data (i.e., the amount of data that can be processed in a single run), and grouped according to this granularity. With this setup, a set of data points sharing a single scaling factor can be processed in one computation cycle, without needing to span multiple computation cycles, thus improving hardware implementation and fully utilizing hardware computing power.

[0103] For example, the number of data points to be quantized in a set of data can be determined based on the number of input ports of the processing circuit performing quantization or the number of input ports of the arithmetic unit that uses quantized data for calculation, and grouped accordingly. With this setup, a set of data to be quantized can be input at once using the number of hardware input ports, allowing a set of data sharing a single scaling factor to be processed in a single input, eliminating the need for multiple inputs. This improves hardware implementation, reduces hardware processing latency, and lowers I / O overhead.

[0104] For example, assuming the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is also 32, then the data points to be quantized can be grouped into sets of 32. If the aforementioned processing circuit or arithmetic unit can process 64 data points in a single operation, and its corresponding number of input ports is less than 64, for example, 32, then the data points to be quantized can be grouped according to the number of input ports, ensuring that the number of data points to be quantized in each group is equal to the number of input ports. In this case, 32 data points are input at once, and the two groups of data can be processed in parallel; alternatively, the 64 data points to be quantized can still be grouped together and input twice to complete the processing of one group of data points at once. If the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is more than 32, then the 32 data points to be quantized can be grouped together to achieve one-time input and processing of one group of data points.

[0105] In some embodiments, the data to be quantized can be elements of tensor data. Dividing multiple data to be quantized into multiple groups can include: dividing the multiple data to be quantized into multiple groups along at least one dimension of the tensor data. For example, suppose tensor data A(B, S), where B and S represent the dimensions of the first and second dimensions, respectively. If the division is performed along the second dimension, taking a grouping granularity of 32 as an example (i.e., each group contains 32 data to be quantized), dividing 32 consecutive data to be quantized along dimension S into a group results in (B×S / 32) groups. Taking each group as a data block, the shape of the tensor composed of each data block after grouping (i.e., the dimensions and dimensions of the tensor) can be represented as (B, S / 32), that is, the dimension of the second dimension after grouping is S / 32.

[0106] For example, suppose tensor data A'(B, S, D), where B, S, and D represent the dimensions of the first, second, and third dimensions, respectively. If we divide the data along the third dimension, taking a grouping granularity of 64 as an example, we can divide 64 consecutive data points to be quantized along the third dimension into a group, resulting in (B×S×D / 64) groups. Each group can be considered as a data block. The shape of the tensor composed of these data blocks after grouping can be represented as (B, S, D / 64), which means the size of the third dimension after grouping is D / 64.

[0107] It is understandable that dividing multiple data sets to be quantized into multiple groups is not limited to dividing them only along one dimension of the tensor data; it can also be done along multiple dimensions as needed. The grouping granularity is not limited to the aforementioned 32 or 64; it can be set as needed, for example, 16, 128, etc. The grouping granularity is related to the computing power or input port limitations of the processing circuit / arithmetic unit mentioned earlier, and will not be elaborated upon here.

[0108] It should be further explained that as the number of groups (i.e., the number of data blocks) increases, the number of scaling factors also increases accordingly. The scaling factors can represent a larger number of data points, which is beneficial for further improving data quantization accuracy. In particular, as the number of scaling factors increases, the sum of the exponent bit width and mantissa bit width increases, thus allowing for the representation of more data points than the E8M0 data format in MX quantization. Consequently, compared to MX quantization, the data accuracy represented by the scaling factors in the extended MX quantization of this disclosed embodiment is higher.

[0109] Therefore, based on the technical solution of obtaining scaling factors through data type conversion and the maximum absolute value, group quantization is used to increase the quantization granularity. A corresponding scaling factor is provided for each group of data to be quantized, which is conducive to further improving the accuracy of the quantized data and the accuracy of the executed task.

[0110] Exemplary device for extended MX quantization

[0111] Figure 5 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure. As shown, the apparatus 500 may include a first processing circuit 510 and a second processing circuit 520, wherein the first processing circuit 510 is used to output a scaling factor, and the second processing circuit 520 is used to perform quantization processing. Specifically, the first processing circuit 510 may be used to: determine the maximum absolute value in a set of data to be quantized; obtain the maximum value that the target data type can represent after quantization of the data to be quantized; determine a scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling factor, wherein the bit width of the second data type is smaller than the bit width of the first data type. The second processing circuit 520 may be used to quantize a set of data to be quantized using the scaling factor to output quantized data with the target data type.

[0112] In some embodiments, the first processing circuit 510 may be further configured to: determine the number of data to be quantized in a set of data to be quantized based on the single processing volume and / or the number of input ports of the second processing circuit 520; or determine the number of data to be quantized in a set of data to be quantized based on the single operation volume and / or the number of input ports of the arithmetic unit that performs operations using quantized data.

[0113] In some embodiments, the apparatus 500 may further include a compression circuit 540. The compression circuit 540 is configured to compress two or more scaling factors after quantization of two or more sets of data to be quantized. In this case, the second processing circuit 520 is configured to quantize the corresponding sets of data to be quantized using the compressed scaling factors, respectively, to output quantized data with the target data type.

[0114] Correspondingly, when device 500 needs to dequantize extended MX quantized data using a compressed format, the scaling factors must first be decompressed. In these embodiments, device 500 may also include a decompression circuit (not shown in the figure). The decompression circuit is configured to decompress the compressed scaling factors in response to the scaling factors being in a compressed format. At this time, the circuit in device 500 that performs the dequantization process (e.g., the second processing circuit 520 or the third processing circuit 530) can be configured to perform dequantization processing on the quantized data using the decompressed scaling factors.

[0115] The operations performed by the first processing circuit 510, the second processing circuit 520, the compression circuit 540, and the decompression circuit have been described in detail above in conjunction with the extended MX quantization method, and will not be repeated here.

[0116] In some embodiments, the first processing circuit 510 may include: a comparison circuit 511, which can be used to determine the maximum absolute value in a set of data to be quantized; a first division circuit 512, which can be used to obtain the maximum value that the target data type after quantization of the data to be quantized can represent, and determine the scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and a first rotation circuit 513, which can be used to convert the scaling factor from a first data type to a second data type to output the scaling coefficient. In some embodiments, the first division circuit 512 can obtain the maximum value by means of table lookup or other methods.

[0117] In some embodiments, the second processing circuit 520 may include a second division circuit 521 and a second rotation circuit 522, wherein the second division circuit 521 can be used to perform a division operation on the data to be quantized and the scaling factor, and the second rotation circuit 522 can be used to perform a rotation operation on the result of the division operation (i.e., the intermediate quantization result) to output the result of quantization of the data to be quantized (i.e., the quantized data).

[0118] In other embodiments, the second processing circuit 520 can also be used to perform dequantization processing on the quantized data using scaling factors. For example, the second processing circuit 520 may also include a multiplication circuit for multiplying the quantized data and the scaling factors. In still other embodiments, the second rotation circuit 522 can also be used to perform rotation operations on the quantized data to convert the quantized data from the target data type to the data type required for dequantization processing; the multiplication circuit is used to multiply the rotated quantized data and the scaling factors to obtain the dequantization result. When the required data type is the original data type of the aforementioned data to be quantized, dequantization processing can restore the quantized data to the data to be quantized before the quantization operation.

[0119] It is understood that the dequantization process is not limited to being performed by the second processing circuit 520 as described above, but can also be performed by other circuits. For example, in some embodiments, the apparatus 500 may further include a third processing circuit 530 (shown in a dashed box in the figure) for dequantizing the quantized data using a scaling factor. The third processing circuit 530 may receive, for example, quantized data output by the second processing circuit 520, and may receive scaling factors output from the first processing circuit 510. In some embodiments, a storage circuit may be further included between the third processing circuit 530 and the second processing circuit 520, or the second processing circuit 520 may include a storage circuit for storing quantized data so that the third processing circuit 530 can read the quantized data from the storage circuit for dequantization processing.

[0120] In some embodiments, when a piece of data to be quantized is input to the comparison circuit 511, the comparison circuit 511 can output the absolute value of that piece of data as the maximum absolute value. In some embodiments, when multiple pieces of data to be quantized are input to the comparison circuit 511, the comparison circuit 511 can output the maximum absolute value among the multiple pieces of data to be quantized. In still some embodiments, the comparison circuit 511 can also be used to divide the multiple pieces of data to be quantized into multiple groups and output the maximum absolute value of each group of data to be quantized; the first division circuit 512 can be further used to determine a scaling factor corresponding to each group of data to be quantized; and the first revolution circuit 513 can be further used to determine a scaling coefficient corresponding to each group of data to be quantized based on each scaling factor. In some embodiments, the data to be quantized is an element of tensor data, and the comparison circuit 511 can be further used to divide the multiple pieces of data to be quantized of the tensor data into multiple groups in at least one dimension of the tensor data.

[0121] It is understood that the above description is exemplary and not restrictive. For example, the first processing circuit for obtaining the scaling factor may not be limited to the comparison circuit, the first division circuit, and the first rotation circuit described above, but may also be implemented using, for example, a lookup table (circuit). Since the dividend (the maximum value that the target data type can represent) is a constant, a lookup table or similar method can be used to directly determine the value based on the input data, thereby implementing the division operation and data type conversion functions described above to obtain the scaling factor. Furthermore, to facilitate understanding of the grouping function implemented by the comparison circuit, an exemplary description will be provided below with reference to Figure 6.

[0122] Figure 6 shows a schematic block diagram of a comparison circuit according to some embodiments of this disclosure. As shown, the comparison circuit 511 may include a comparator tree composed of multiple levels of comparators, for example, it may include layers a, b, c, ..., z, each level including one or more comparators, with the number of comparators decreasing sequentially at each level. The outputs of multiple comparators in the upper level of an adjacent level can be input to the comparators in the lower level. In practical applications, the outputs of the required levels can be selected as needed by means of software or hardware to achieve grouping of multiple data to be quantized and to obtain the maximum absolute value of each group of data to be quantized. For ease of understanding, further examples will be provided below with reference to Figure 6.

[0123] As further illustrated in Figure 6, layer a includes comparators a1, a2, a3, ..., an, where n represents the number of comparators in layer a; layer b includes comparators b1, b2, etc.; layer c includes comparator c1, etc.; and layer z includes comparator z1. It is understood that the comparator tree shown in the figure is exemplary, and the number of levels in the comparator tree can be set more or less as needed, and the number of comparators included in each level can also be set as required.

[0124] In some embodiments, the number of comparators *n* in layer *a* can be the same as the number of input data to be quantized, so that each data point to be quantized is input into a corresponding comparator in layer *a*. As the number of comparators in subsequent layers decreases sequentially, the output can be gradually reduced to obtain the desired grouping result and the maximum absolute value of each group. For example, assuming the input layer receives 128 data points to be quantized, and layer *a* has 128 comparators, then each data point to be quantized is input into a corresponding comparator in layer *a*. The output of layer *a* can then divide the 128 data points to be quantized into 128 groups. Assuming layer *b* includes 64 comparators, then the output of every two comparators in layer *a* can be input into a comparator in the next layer, layer *b*. For example, the outputs of comparators *a1* and *a2* in layer *a* can be input into comparator *b1* in layer *b*. Layer *b* then outputs 64 maximum absolute values, thus dividing the 128 data points to be quantized into 64 groups. Assuming layer c contains 32 comparators, the outputs of every two comparators in layer b can be input into a comparator in the next layer c. For example, the outputs of comparators b1 and b2 in layer b can be input into comparator c1 in layer c. At this point, layer c outputs 32 maximum absolute values, effectively dividing the 128 data points to be quantized into 32 groups. This process continues until the last layer z outputs a single maximum absolute value, thus dividing the 128 data points to be quantized into one group.

[0125] Taking this embodiment as an example, in practical applications, assuming that 128 data points to be quantized need to be divided into groups of four, resulting in 32 groups, the output of layer C can be selected as the output of comparator circuit 411 through methods such as programming, logic control, or selectors. This allows the first division circuit to determine the scaling factor corresponding to each of the 32 groups of data to be quantized based on the 32 outputs of layer C (i.e., the 32 maximum absolute values). Therefore, the comparator tree provides multiple granularity grouping methods, allowing users to flexibly select the output of a certain level in the comparator tree according to the required quantization granularity.

[0126] Furthermore, it should be noted that the comparator tree structure is not limited to the two comparators in each level shown in the diagram whose outputs correspond to one comparator in the next level. It can also be configured to have, for example, three or more comparators whose outputs correspond to one comparator in the next level. For instance, assuming level a has 128 comparators, level b can have 4 comparators. That is, the outputs of the 32 comparators in level a correspond to one comparator in level b. Therefore, the output of level b can divide the 128 data points to be quantized into 4 groups and output the maximum absolute value of the 32 data points in each group.

[0127] In other embodiments, the number of comparators n in layer a can be different from the number of input data to be quantized, as illustrated below. Assuming the number of comparators in layer a is less than the number of data to be quantized, multiple data to be quantized can be input into a single comparator in layer a. This achieves the division of multiple data to be quantized into fewer groups for output in layer a. In this embodiment, the data to be quantized input to each comparator in layer a can also be controlled before layer a (e.g., in the input layer) by using methods such as address decoders, multiplexers (MUX), or other logic circuits.

[0128] For example, suppose the input layer receives 128 data points to be quantized, and layer 'a' has 64 comparators. Then, every two data points to be quantized are input into one of the corresponding comparators in layer 'a'. At this point, the output of layer 'a' divides the 128 data points into 64 groups. The number of comparators in subsequent layers is halved sequentially until the last layer outputs the maximum absolute value, at which point the 128 data points are divided into one group.

[0129] The foregoing description, in conjunction with Figure 6, exemplarily illustrates the scheme for implementing grouping and outputting the maximum absolute value using a comparison circuit according to an embodiment of this disclosure. It is understood that the structure of the comparison circuit shown in the figure and the preceding description are exemplary and not restrictive. For example, the last level of the comparison circuit 511 may not be limited to including only one comparator as shown in the figure, but may also include multiple comparators as needed, such as using level c in the figure as the last level. Furthermore, the comparator circuit 511 may not be limited to including only one comparator tree, but may also include multiple comparator trees as needed, thereby enabling, for example, parallel grouping. The comparison circuit of this embodiment of the disclosure may not be limited to being implemented using the aforementioned comparator tree, but may also be implemented based on radix sorting or other methods.

[0130] It is also understood that the grouping method in this disclosed embodiment is not limited to the comparison circuit, but can also be implemented by combining software. For example, by setting grouping parameters (including information such as the number of data to be quantized in each group K) through software, the comparison circuit can read K consecutive data to be quantized as a group according to these grouping parameters, and directly output the maximum absolute value in each group of data to be quantized.

[0131] In this disclosed embodiment, when referring to "particle quantization format," "particle size format," or "particle quantization," it refers to the three-part data format shown in Figure 4. However, the specific element data types, scaling block sizes, scaling factor data types, etc., are not limited to those listed in Table 1, but can include many other combinations. In other words, the particle quantization format can include the MX quantization format proposed by OCP, the extended MX quantization format proposed in this disclosed embodiment, or any other data format conforming to the structure of Figure 4.

[0132] In this disclosed embodiment, the term "instruction" can include software instructions, hardware instructions, firmware instructions, or any combination thereof. Software instructions typically refer to statements or commands in a programming language; they are high-level, abstract instructions, such as function calls, machine code, and bytecode. Hardware instructions, also known as machine instructions or instruction set architecture (ISA) instructions, are low-level commands that computer hardware can directly recognize and execute, such as processor instructions and CPU instructions. Firmware instructions include, for example, opcodes and microcode. Furthermore, different CPU architectures and different instruction sets can have different hardware instruction sets. Based on the complexity of the computer instruction set, hardware design, execution speed, compiler complexity, and instruction format, they can be further classified as: CISC (Complex Instruction Set Computer), RISC (Reduced Instruction Set Computer), and VLIW (Very Long Instruction Word).

[0133] Exemplary data processing device

[0134] This disclosure proposes a vector granular quantization / dequantization scheme. This scheme integrates vector operations with granular quantization / dequantization. For example, the input vector is first quantized granularly, and then the quantization result is processed; or the input vector is first dequantized granularly, and then the dequantized result is processed. Alternatively, operations between input vectors are performed first, and then the result is quantized granularly; or operations between input vectors are performed first, and then the result is dequantized granularly. By integrating the granular quantization or dequantization process of the input vector into the vector operations, separate quantization / dequantization processing is avoided, reducing overhead such as memory access in the entire process and optimizing performance.

[0135] Figure 7 illustrates a schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure. It can be understood that this structure can be viewed as a refinement of the internal structure of a single processing core in Figure 3, or as a functional partitioning diagram based on multiple processing cores shown in Figure 3. As shown in Figure 7, the data processing apparatus 700 may include a control circuit 710, a storage circuit 720, and an arithmetic circuit 730, which are interconnected to transmit various data and instructions.

[0136] The control circuit 710 is responsible for handling various functions on the data processing device 700, including but not limited to control, instruction fetching, decoding, and calculation. The control circuit 710 may include, for example, the control module 31 shown in FIG3.

[0137] The storage circuit 720 can be used to store information, including at least pre-processing and / or post-processing information, and may also include intermediate information that needs to be cached during processing, such as various RAMs shown in FIG3, or on-chip caches. In some embodiments, the storage circuit 720 can be configured to store input vectors, vector operation results, and / or cache intermediate results, and may include, for example, a first storage circuit 721 for storing the input vector to be operated / quantized / dequantized and a second storage circuit 722 for storing the operation / quantization / dequantization results. Of course, those skilled in the art will understand that the first storage circuit 721 and the second storage circuit 722 can physically be the same storage circuit.

[0138] The arithmetic circuit 730 can be configured to perform various operations according to vector operation instructions. For example, the arithmetic circuit 730 can be configured to, under the control of the control circuit 710, perform particle quantization or dequantization on the input vector and then perform operations, or perform operations on the input vector and then perform particle quantization or dequantization on the operation result. In this case, the arithmetic circuit 730 may include a vector operation quantization circuit 731, configured to perform particle quantization or dequantization on the input vector data and then perform operations, or perform operations on the input vector data and then perform particle quantization or dequantization on the operation result.

[0139] Vector operations can include one or more of the following: addition and subtraction, comparison, logical operations, shift operations, accumulation, transcendental functions, rounding, table lookup, generating an increasing sequence, and bit-by-bit extension.

[0140] Addition and subtraction operations: Add or subtract corresponding elements of two vectors.

[0141] Comparison operation: compares the elements of two vectors, usually producing a Boolean vector indicating whether the element at each position satisfies a certain comparison condition (such as equal to, greater than, etc.).

[0142] Logical operations: Perform logical operations (such as AND, OR, NOT) on the elements of a vector.

[0143] Shift operation: Moves the elements of a vector to the left or right by a certain number of positions, and can fill the new positions with zero or other values.

[0144] Accumulation operation: Accumulates the elements in a vector, either element-by-element or across multiple elements.

[0145] Transcendental functions: Apply a transcendental function, such as an exponential function, a logarithmic function, or a trigonometric function, to each element of a vector.

[0146] Rounding: Performs a rounding operation on each element of a vector, which can be rounding down, rounding up, or rounding to the nearest integer.

[0147] Rounding: Rounds each element of a vector, for example, to the nearest integer or to a specified number of decimal places.

[0148] Lookup table: Use a lookup table (LUT) to map each element of a vector to another value.

[0149] Generate an increasing sequence: Generate a vector whose elements are an increasing sequence, such as from 1 to N.

[0150] Bit-by-bit expansion: Extend each element of the vector bit by bit, for example, extend 8 bits to 16 bits.

[0151] In some embodiments, fused vector operations with multiple operands can be supported. For example, not only can addition and subtraction operations between input vector A and input vector B be supported, but also AND logical operations between input vector B and input vector C can be performed first, and the result can be added to input vector A, i.e., A+(B AND C) operation. In other words, one vector operation instruction can complete multiple operations.

[0152] In some embodiments, the arithmetic circuit 730 further includes: a read circuit 732 configured to read an input vector from the first storage circuit 721 and provide it to the vector arithmetic quantization circuit 731; and a write circuit 733 configured to write the result of particle quantization or dequantization performed by the vector arithmetic quantization circuit 731 to the second storage circuit 722.

[0153] The vector operation quantization circuit 731 performs particle quantization or dequantization on the input vector, or particle quantization or dequantization on the result of the operation on the input vector. For details, please refer to the description in the previous section on exemplary extended MX quantization methods, which will not be described here again.

[0154] In some embodiments, the scaling factors obtained by extended MX quantization can adopt the aforementioned compression format to save storage space and transmission bandwidth. In this case, the vector operation instruction can be supplemented with corresponding indication information to indicate this compression format of the scaling factors. The vector operation quantization circuit 731 may also include a decompression circuit (not shown), configured to decompress the compressed scaling factors in response to the vector operation instruction indicating that the scaling factors are in a compressed format, to obtain decompressed scaling factors. In some implementations, the compression format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. In this case, the vector operation quantization circuit 731 is further configured to combine the shared exponent with the Z new mantissas respectively to obtain Z decompressed scaling factors for subsequent use, such as in computation or dequantization.

[0155] In some embodiments, the control circuit 710 is further configured to segment the vector operation instructions to obtain micro-instructions, and the operation circuit 730 is configured to perform particle quantization or dequantization on the input vector according to the micro-instructions and then perform the operation, or perform the operation on the input vector and perform particle quantization or dequantization on the operation result.

[0156] By breaking down a vector operation instruction into multiple micro-instructions, various situations can be handled more flexibly during execution, supporting more advanced scheduling techniques to optimize the execution order of instructions, reduce resource conflicts, and so on.

[0157] The scale of micro-instruction segmentation for vector operation instructions can be determined based on one or more of the information in the data processing device 700, such as the transmission bandwidth, the size of the short input vector and / or long input vector data blocks in the input vector.

[0158] In some embodiments, the vector operation instructions are also used to perform a reduction operation on the input vector or its operation result. In this case, the operation circuit 730 can also be configured to perform a reduction operation on the input vector operation result under the control of the control circuit 710. In some embodiments, the reduction operation is performed before quantizing the input vector operation result. Of course, the reduction operation can also be performed after quantizing the input vector operation result.

[0159] A reduction operation is an operation that combines multiple elements in a vector or array into a single value. This can be achieved by applying a function, which can be any of the following.

[0160] `accsum` (accumulated sum): Adds all elements of a vector together to get a single numerical value. For example, if you have a vector [1,2,3,4], then its accumulated sum is 1+2+3+4=10.

[0161] `max(maximum value)`: Finds the maximum element in a vector. For the vector [1,3,2,4], the maximum value is 4.

[0162] min(minimum): Finds the minimum element in a vector. For the same vector [1,3,2,4], the minimum value is 1.

[0163] Product: Calculates the product of all elements in a vector.

[0164] Mean / average: Calculates the average value of all elements in a vector.

[0165] The sum of squares root: Calculates the square root of the sum of the squares of all elements in a vector.

[0166] Logical AND: The result is true if all elements in the vector are true (in a Boolean context).

[0167] Logical OR: The result is true if at least one element in the vector is true.

[0168] Bitwise AND: Performs an AND operation on the corresponding bits of all elements in a vector.

[0169] Bitwise OR: Performs an OR operation on the corresponding bits of all elements in a vector.

[0170] Bitwise XOR: Performs an XOR operation on the corresponding bits of all elements in a vector.

[0171] According to this embodiment, since the reduction operation is also integrated with the quantization / dequantization of the computation result, the number of memory accesses can be reduced, thereby reducing latency and improving processing speed. In particular, as mentioned earlier, during particle quantization, it is necessary to determine the maximum absolute value in a set of data to be quantized. Therefore, by integrating the maximum value reduction operation with the particle quantization operation, processing efficiency can be significantly improved.

[0172] It is understandable that when the reduction operation is also integrated with the quantization of the computation result, the vector operation quantization circuit 731 needs to output three results simultaneously, including the scaling factor, the revolution result, and the reduction result.

[0173] In some embodiments, when a vector operation instruction instructs the input vector to be dequantized at a fine-grained level and then the operation is performed, the read circuit is configured to read scaling factor data for a specific number of cycles and the input vector data to be dequantized for the corresponding number of cycles in response to the vector operation instruction; the vector operation quantization circuit is configured to perform fine-grained dequantization based on the read scaling factor data and the input vector data and perform the operation.

[0174] When performing fine-grained dequantization, the vector operation instructions add a memory access requirement for the scaling factor X.

[0175] In current AI chips, memory access to on-chip RAM is often a bottleneck, as existing data already consumes most of the on-chip memory bandwidth. To address the memory access requirements of newly added scaling factors, two approaches can be taken. One approach is to increase memory access bandwidth, for example, by adding several new ports to the on-chip RAM for accessing these new scaling factors. The other approach is to reuse existing ports for scaling factor access. Similarly, a selector can be added to the existing output write-back port to select whether to write back the result data or the result scaling factor.

[0176] When multiplexing ports, data cannot be continuously supplied because time needs to be allocated for reading scaling factors, which inevitably leads to some performance degradation. Therefore, in some embodiments, the property that scaling factors are shared within granular blocks can be utilized to fully utilize the bandwidth when reading scaling factors in each cycle, and buffer them for use in subsequent cycles.

[0177] Specifically, in some embodiments, the vector operation instructions also instruct the scaling factor and its corresponding data to reuse the same read port, and for each frame of scaling factor read, multiple frames of corresponding data are read proportionally. It can be understood that, even without port reuse, the scaling factor read port and the data read port can still be controlled to perform read operations proportionally. For example, for each frame of scaling factor read by the scaling factor read port, the data read port reads the corresponding data for n frames. In this way, the scaling factor read port will be idle for n-1 frames, which can then be used to read other data.

[0178] In these embodiments, as shown in FIG7, the data processing apparatus 700 further includes a first buffer 741 configured to buffer scaling factors read from off-chip to be supplied in proportion to the corresponding data to the vector operation quantization circuit 731 in the arithmetic circuit 730 for operation.

[0179] The above ratio is determined at least based on the granularity block size T, the data bit width of the scaling factor, and the data bit width of the corresponding data. Since a scaling factor is shared among K (i.e., T = K, e.g., 32) elements, the ratio is 1 scaling factor to T corresponding data. This is based on the data bit width X of the scaling factor. bw and the corresponding data bit width P bw The usage ratio (in bytes) between the two can be determined to be: 1×X bw :T×P bw .

[0180] For example, when the scaling factor is in E8M3 format with a bit width of 2 bytes, and the element data is in FP8 format with a bit width of 1 byte, and T = 32, the ratio is 2:32 = 1:16. When reading data in each frame, whether reading the scaling factor or the corresponding data, the amount of data read should be maximized to fill the transmission bandwidth, thus fully utilizing transmission performance. For example, assuming a transmission bandwidth of 128 bytes, each frame of scaling factor reading (128 bytes / 2 bytes = 64 scaling factors) corresponds to reading 16 frames of element data (128 bytes × 16 / 1 byte = 64 × 32). The read scaling factors are buffered in the first buffer. When the arithmetic circuit performs calculations, for every 128 bytes of element data supplied (e.g., filling the entire arithmetic bandwidth), 128 bytes / 16 = 8 bytes of scaling factors are supplied from the first buffer. Therefore, the scaling factor read in one frame can remain in the first buffer for multiple frames, thus alleviating bandwidth constraints.

[0181] It is understandable that when the scaling factor adopts a compressed format, the above ratio can be adjusted accordingly based on the compression ratio. For example, when the scaling factor format is E8M3, every two adjacent scaling factors are compressed, occupying 2 bytes after compression. The element data is in FP8 format, with a bit width occupying 1 byte. When T=32, the above ratio is 2:2×32=1:32. It can be seen that reading 2 bytes of scaling factor corresponds to 64 elements of data, requiring the reading of 64 bytes of element data. The vector operation quantization circuit 731 can read the compressed scaling factor from the first buffer 741, decompress it, and perform the corresponding operation.

[0182] Similarly, when the output of vector operations requires granular scaling, the write-back memory circuit will also include the result data and the result scaling factor. Likewise, since a result scaling factor is shared among K (e.g., 32) result data, there may not be enough scaling factors when the result data can fill the write-back transmission bandwidth. To control the efficient use of the write-back path's transmission bandwidth and avoid write port bottlenecks, a second buffer (second buffer 742 in Figure 7) can be set up to temporarily store the result scaling factors. When the result scaling factors accumulate to a certain amount, such as when the transmission bandwidth is full, the write-back task is executed, writing these result scaling factors back to the memory circuit in one cycle. Therefore, the output order can be controlled to output n cycles of result data first, followed by one cycle of result scaling factors. Similarly, n is determined based on the granularity block size T, the bit width of the result data, and the bit width of the result scaling factor.

[0183] The granular quantization of the vector operation output can also be performed in the vector operation quantization circuit 731 in Figure 7. In some embodiments, the vector operation quantization circuit 731 may further include a compression circuit (not shown), configured to: compress two or more scaling factors after quantizing two or more sets of data to be quantized, in response to a vector operation instruction indicating that the scaling factor is in a compressed format. In this case, the vector operation quantization circuit 731 uses the compressed scaling factors to quantize the corresponding sets of data to be quantized, respectively, to output quantized data with the target data type. Similarly, when the scaling factor is in a compressed format, the output ratio during write-back is adjusted accordingly based on the compression ratio.

[0184] In some embodiments, the vector operation instruction can be a first cycle-type instruction, also known as a low-dimensional cycle instruction. In this instruction mode, the two input vectors have different lengths; the shorter one is called the short input vector, and the longer one is called the long input vector. In this instruction mode, vector operations are performed as follows: the short input vector data is copied N times and then operated on with the long input vector, where the length of the long input vector is N times the length of the short input vector, and N is a positive integer greater than 1. That is, under normal circumstances, the read circuit 732 needs to repeatedly read the short input vector from the first storage circuit 721 N times, perform operations with the long input vector, and finally obtain an operation result with the length of the long input vector.

[0185] Therefore, in some embodiments, to reduce the number of times the read circuit 732 reads the short input vector from the first storage circuit 721, thereby improving performance, the short input vector can be read into the first buffer 741 for caching, and data can be retrieved from the first buffer 741 for computation when needed. When the short input vector is short and its data size is smaller than the storage capacity of the first buffer 741, the read circuit 732 reads all the short input vector data into the first buffer 741. When the short input vector is long, resulting in its data size being larger than the storage capacity of the first buffer 741, for example, when the data size of the short input vector is twice the storage capacity of the first buffer 741, the read circuit 732 first reads a first portion of the short input vector data, for example, short input vector data the size of the first buffer, into the first buffer 741. In this case, the vector operation quantization circuit 731 copies the specific number of short input vector data (the first part of the short input vector data of the size of the first buffer) N times, performs the operation with the long input vector, and performs particle quantization on the operation result, where the length of the long input vector is N times the length of the short input vector, and N is a positive integer greater than 1.

[0186] After completing the operations on the first part of the short input vector data and the long input vector, and performing granular quantization on the operation results, the reading circuit 732 reads the second part of the short input vector data into the first buffer 741. The vector operation quantization circuit 731 copies the read second part of the short input vector data N times, performs operations with the long input vector, and performs granular quantization on the operation results.

[0187] When a short input vector is copied N times in parts, and then processed with a long input vector, and the results are quantized, it is necessary to provide the scaling factor and the actual correspondence between the data.

[0188] For example, if the input vector data type is FP16, the scaling factor data type is FP8, the short input vector size is 2KB, the first buffer 741 size is 1KB, and the short input vector length is 2024*8 / 16 = 1024, then with a granularity block size of 32, this corresponds to 1024 / 32 = 32 scaling factors. The long input vector length is 8 times that of the short input vector. Therefore, in the case of a low-dimensional cycle instruction, the length of the resulting operation should be 1024*8 = 8192, generating a total of 8192 / 32 = 256 scaling factors. The scaling factors generated from the first part of the short input vector data actually correspond to scaling factors 0-15, 32-47, 64-79, 96-111, 128-143, 160-175, 192-207, and 224-239.

[0189] Furthermore, in some embodiments, when the vector operation instruction is a first cycle type instruction, since the short input vector data needs to be copied N times before being operated on with the long input vector, if the length of the long input vector is not a positive integer multiple of the length of the short input vector, the vector operation quantization circuit 731 will also pad the short input vector upward to one of the positive integer multiples of the long input vector, and thus discard invalid revolution results and scaling factors, which are generated by the data used to pad the short input vector to one of the positive integer multiples of the long input vector.

[0190] In some embodiments, the vector operation instruction can be a second cycle type instruction, also known as a high-dimensional cycle instruction. In this instruction mode, the two input vectors have different lengths; the shorter one is called the short input vector, and the longer one is called the long input vector. In this instruction mode, the vector operation is performed as follows: each element in the short input vector is copied M times and then operated on with the long input vector, where the long input vector has M elements.

[0191] In some embodiments, similar to the measures taken above for the low-dimensional cycle instruction mode, in order to reduce the number of times the read circuit 732 reads short input vector elements from the first storage circuit 721 and thus improve performance, the short input vector can be read into the first buffer 741 for caching. When needed, data can be retrieved from the first buffer 741 for computation. That is, the read circuit 732 can read a specific number of short input vector data into the first buffer 741. The vector operation quantization circuit 731 copies each element in the specific number of short input vector data read M times, performs the operation with the long input vector, and performs granular quantization on the operation result, wherein the long input vector has M elements. After using each element of a specific number of short input vector data, the read circuit 732 can continue to read another specific number of short input vector data into the first buffer 741. The vector operation quantization circuit 731 continues to copy each element in the read another specific number of short input vector data M times, and then perform operations with the long input vector, and perform granular quantization on the operation result, until each element in all short input vectors has been copied M times and operated with the long input vector, to obtain an operation result with the length of the product of the long input vector length and the short input vector length.

[0192] In some embodiments, when the vector operation instruction is a first cycle type instruction and also instructs to perform a reduction operation on the operation result of the input vector, the vector operation quantization circuit 731 generates a reduction result for each short input vector, and generates a total of N reduction results, where N is a multiple of the length of the long input vector relative to the length of the short input vector, that is, the operation result with the length of the long input vector is divided by the length of the short input vector to obtain N parts of the operation result, and each part of the operation result generates one reduction result.

[0193] In some embodiments, when the vector operation instruction is a second cycle type instruction and also instructs to perform a reduction operation on the operation result of the input vector, the vector operation quantization circuit 731 generates a reduction result for each long input vector, and generates a total of P reduction results, where P is the number of elements of the short input vector, that is, the operation result with the length of the product of the long input vector length and the short input vector length is divided by the long input vector length to obtain P partial operation results, and each partial operation result generates 1 reduction result.

[0194] In other embodiments, regardless of the type of vector operation instruction, the operation results that produce the reduction result can be divided according to the size of the granularity block used in the particle quantization. For example, when the granularity block size is 32, then one reduction result can be generated for every 32 operation result elements.

[0195] The vector operation scheme with fused particle quantization has been described above with reference to Figure 7. Those skilled in the art will understand that although the various buffers and operation circuits are shown as separate modules in Figure 7, depending on the configuration, the buffers and operation circuits can also be combined into a single module. For example, the first and second buffers can be combined with the vector operation quantization circuit. The embodiments disclosed herein are not limited in this respect.

[0196] Figure 8 shows an exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of this disclosure. In this data processing method, a control circuit controls an arithmetic circuit to perform operations on an input vector.

[0197] More specifically, in step 810, the control circuit parses vector operation instructions, which are used to perform particle quantization or dequantization on the input vector and then perform the operation, or to perform the operation on the input vector and then perform particle quantization or dequantization on the operation result.

[0198] Next, in step 820, under the control of the control circuit, the arithmetic circuit performs particle quantization or dequantization on the input vector according to the parsed vector arithmetic instruction and then performs the arithmetic, or performs the arithmetic on the input vector and performs particle quantization or dequantization on the arithmetic result.

[0199] Those skilled in the art will understand that the steps described in the method flowchart correspond to the various circuits of the data processing device described above in conjunction with Figure 7, and therefore the features described above also apply to the method steps, and will not be repeated here.

[0200] This disclosure also provides a processor, including the aforementioned data processing apparatus. This disclosure further provides a chip, which may include the processor of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a board that may include the aforementioned chip.

[0201] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0202] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this disclosure, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0203] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0204] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately. For example, in some embodiments, the read circuit and the write circuit may be integrated into the vector operation quantization circuit.

[0205] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM (read-only memory), and RAM (random access memory), etc.

[0206] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A data processing apparatus, comprising: The control circuit is configured to parse vector operation instructions, which instruct either to perform particle quantization or dequantization on an input vector before performing the operation, or to perform the operation on the input vector and then perform particle quantization or dequantization on the result. as well as The arithmetic circuit is configured to perform operations on the input vector after particle quantization or dequantization according to the vector operation instruction, or to perform operations on the input vector and then perform particle quantization or dequantization on the operation result.

2. The data processing apparatus according to claim 1, wherein, The arithmetic circuit includes: A read circuit, configured to read an input vector from a first storage circuit; A vector operation quantization circuit is configured to perform particle quantization or dequantization on a read input vector before performing operations, or to perform operations on a read input vector and perform particle quantization or dequantization on the operation result; and The write circuit is configured to write the result produced by the vector operation quantization circuit to the second storage circuit.

3. The data processing apparatus according to claim 2, wherein, The vector operation quantization circuit is configured to perform extended MX quantization of the vector data to be quantized through the following operations: Determine the maximum absolute value in a set of vector data to be quantized; Obtain the maximum value that the target data type can represent after the quantized vector data is quantized; Based on the ratio between the maximum absolute value and the maximum value, the scaling factor corresponding to the set of vector data to be quantized is determined; The scaling factor is converted from a first data type to a second data type to output a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type; as well as The scaling factor is used to quantize the set of vector data to be quantized, so as to output quantized data with the target data type.

4. The data processing apparatus according to claim 3, wherein, The vector operation quantization circuit is further configured to: In response to the vector operation instruction indicating that the scaling factor is in a compressed format, the two or more scaling factors after quantization of two or more sets of vector data to be quantized are compressed; as well as The corresponding groups of data to be quantized are quantized using the compressed scaling factor to output quantized data with the target data type.

5. The data processing apparatus according to claim 4, wherein the vector operation quantization circuit is further configured to: Determine the shared exponent of the Z scaling factors to be compressed, Z≥2; Determine the Z new mantissas corresponding to the Z scaling factors; and The shared index is bundled with the Z new tail numbers and output for association.

6. The data processing apparatus according to claim 5, wherein the vector operation quantization circuit is further configured to: Set the largest index among the Z scaling factors as the shared index.

7. The data processing apparatus according to claim 6, wherein the vector operation quantization circuit is further configured to: Set the new mantissa of the scaling factor whose exponent is less than the maximum exponent to 0; and Set the new mantissa, whose exponent is equal to the scaling factor of the maximum exponent, as the original mantissa.

8. The data processing apparatus according to any one of claims 5-7, further comprising one or more of the following: The second data type of the Z scaling factors to be compressed is the same; The Z scaling factors belong to Z adjacent groups of data to be quantized; Z can be 2, 4, or 8.

9. The data processing apparatus according to claim 2, wherein, The read circuit is configured to read scaling factor data for a specific number of cycles and input vector data to be dequantized for the corresponding number of cycles in response to the vector operation instruction; the vector operation quantization circuit is configured to perform fine-grained dequantization based on the read scaling factor data and input vector data and perform the operation.

10. The data processing apparatus according to any one of claims 2-9, wherein, The vector operation quantization circuit is configured to dequantize extended MX quantized data through the following operations: In response to the vector operation instruction indicating that the scaling factor is in compressed format, the compressed scaling factor is decompressed to obtain the decompressed scaling factor. as well as The dequantization is performed on the extended MX quantized data using the decompressed scaling factor.

11. The data processing apparatus according to claim 10, wherein, The compression format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. The vector operation quantization circuit is further configured for: The shared index is combined with the Z new mantissas to obtain Z decompressed scaling factors for use in the inverse quantization.

12. The data processing apparatus according to claim 2, wherein, When the vector operation instruction is a first cycle type instruction, the reading circuit is configured to read a specific number of short input vector data and a long input vector. The vector operation quantization circuit is used to copy the specific number of short input vector data read N times as a whole, perform operations with the long input vector, and perform particle quantization on the operation result. The length of the long input vector is N times the length of the short input vector, and N is a positive integer greater than 1.

13. The data processing apparatus according to claim 2, wherein, When the vector operation instruction is a second cycle type instruction, the reading circuit is configured to read a specific number of short input vector data and a long input vector. The vector operation quantization circuit is configured to copy each element of the specific number of short input vector data read M times, perform operation with the long input vector, and perform particle quantization on the operation result. The length of the long input vector is greater than the length of the short input vector, and the number of elements of the long input vector is M.

14. The data processing apparatus according to claim 2, wherein, When the vector operation instruction is a first cycle type instruction, and the length of the long input vector is not a positive integer multiple of the length of the short input vector, the vector operation quantization circuit is configured to pad the short input vector upward to one of the positive integer multiples of the long input vector, and is configured to discard invalid revolution results and scaling factors, which are generated by data used to pad the short input vector to one of the positive integer multiples of the long input vector.

15. The data processing apparatus according to claim 12, wherein, The vector operation quantization circuit is also configured to simultaneously provide the actual correspondence between the scaling factor and the data when outputting the scaling factor.

16. The data processing apparatus according to claim 2, wherein, The vector operation instruction also instructs a reduction operation on the result of the operation on the input vector, and the vector operation quantization circuit is further configured to perform a reduction operation on the result of the operation on the input vector.

17. The data processing apparatus according to claim 16, wherein, When the vector operation instruction is a first cycle type instruction, a reduction result is generated for each short input vector, and a total of N reduction results are generated, where N is the multiple of the length of the long input vector relative to the length of the short input vector.

18. The data processing apparatus according to claim 16, wherein, When the vector operation instruction is a second cycle type instruction, a reduction result is generated for each long input vector, and a total of P reduction results are generated, where P is the number of elements in the short input vector.

19. The data processing apparatus according to any one of claims 1-18, wherein, The original data type of the input vector to be quantized, or the result of the operation to be quantized, includes at least one of FP16, BF16, FP32, TF32, and FP64, and the target data type for quantization includes at least one of FP8, FP6, FP4, FP2, int8, int4, int2, and int6, wherein FP8, FP6, and FP4 support multiple data formats; and the original data type of the input vector to be dequantized, or the result of the operation to be dequantized, includes at least one of FP8, FP6, FP4, FP2, int8, int4, int2, and int6, wherein FP8, FP6, and FP4 support multiple data formats, and the target data type for dequantization includes at least one of FP16, BF16, FP32, TF32, and FP64.

20. A processor comprising the data processing apparatus as described in any one of claims 1-19.

21. A board comprising the processor according to claim 20.

22. A data processing method implemented by a data processing device, the data processing device comprising a control circuit and an arithmetic circuit, the method comprising: The control circuit parses vector operation instructions, which are used to perform particle quantization or dequantization on the input vector and then perform the operation, or to perform the operation on the input vector and then perform particle quantization or dequantization on the operation result. The operation circuit performs particle quantization or dequantization on the input vector and then performs the operation, or performs the operation on the input vector and then performs particle quantization or dequantization on the operation result, according to the parsed vector operation instructions.

Images