Data processing apparatus, processor, board, and data processing method

Abstract

The present application discloses a data processing apparatus, a processor, a board, and a data processing method. The data processing apparatus can be comprised in a computing apparatus of a combined processing apparatus, and the combined processing apparatus can further comprise an interface apparatus and another processing apparatus. The data processing apparatus interacts with the other processing apparatus to jointly complete a computing operation specified by a user. The combined processing apparatus can also comprise a storage apparatus. The storage apparatus is separately connected to the computing apparatus and the other processing apparatus, and is used for storing data of the computing apparatus and the other processing apparatus. The present application provides a convolution operation scheme compatible with various fine-grained quantization formats, which can simplify processing and make full use of the advantages of low bit width operation and small memory occupation of the fine-grained quantization formats.

Description

Data processing apparatus, processor, board card and data processing method Cross-reference to related applications

[0001] The present application claims priority from Chinese Patent Application No. 202410813862.X, filed on June 21, 2024, entitled “Data processing apparatus, processor, board card and data processing method” and Chinese Patent Application No. 202411347002.8, filed on September 25, 2024, entitled “Data processing apparatus, processor, board card and data processing method”. TECHNICAL FIELD

[0002] The present disclosure relates generally to the field of computer processors. More specifically, the present disclosure relates to a data processing apparatus, processor, board card and data processing method implemented by the data processing apparatus. BACKGROUND

[0003] With the rapid development of artificial intelligence, the amount of computation and the number of parameters of neural network models are continuously increasing. Studies have shown that during the training and inference process, the running speed, power consumption and chip area can be effectively improved by reducing the precision of floating point operations.

[0004] To this end, AI vendors have designed various floating point formats, such as BF16, TF32, etc., to achieve a balance between AI system performance and precision. These floating point formats all have their own application scenarios and advantages, but the variety of types or special floating point formats also face the problem of migration, for example, a model trained by FP6 of vendor A cannot run on the inference hardware of vendor B.

[0005] In order to form a unified industry standard, OCP (Open Compute Project) has introduced a MX (MicroXcaling) quantization format. The MX quantization format is an interoperable quantization data format with shared, fine-grained block sizes. The MX-compliant data format supports AI training and inference, has lower bit-width arithmetic operations and smaller memory footprint, and thus promotes the improvement of hardware performance and efficiency, thereby reducing overhead and operating costs.

[0006] In view of this, it is necessary to develop an operation mode and corresponding hardware compatible with the MX quantization format for the MX quantization format, especially the convolution operation which accounts for a large proportion in neural network operation, so as to fully exert the advantages of the MX quantization format. SUMMARY

[0007] In order to at least solve one or more technical problems as mentioned above, the present disclosure proposes a convolution operation scheme compatible with the MX quantization format in multiple aspects. Further, an improved MX quantization scheme is also provided.

[0008] In a first aspect, the present disclosure provides a data processing apparatus, comprising: a control circuit configured to parse a convolution instruction, the convolution instruction indicating a convolution operation on an input neuron and a weight to obtain an output neuron, wherein either or both of the input neuron and the weight is in a micro-quantized format; and an operation circuit configured to perform the convolution operation on data of the input neuron and data of the weight according to the convolution instruction, and fuse a scaling factor of the input neuron and / or a scaling factor of the weight in the convolution operation.

[0009] In a second aspect, the present disclosure provides a processor comprising the data processing apparatus of the preceding first aspect.

[0010] In a third aspect, the present disclosure provides a board card comprising the processor of the preceding second aspect.

[0011] In a fourth aspect, the present disclosure provides a data processing method implemented by a data processing apparatus, the data processing apparatus comprising a control circuit and an operation circuit, the method comprising: parsing, by the control circuit, a convolution instruction, the convolution instruction indicating a convolution operation on an input neuron and a weight to obtain an output neuron, wherein either or both of the input neuron and the weight is in a micro-quantized format; and performing, by the operation circuit, the convolution operation on data of the input neuron and data of the weight according to the convolution instruction, and fusing a scaling factor of the input neuron and / or a scaling factor of the weight in the convolution operation.

[0012] In the convolution operation scheme provided by the embodiments of the present disclosure, when either or both of the input neuron and the weight is in a micro-quantized format, the dequantization process of data can be fused in the convolution operation without the need for special dequantization of data in advance, thereby avoiding separate dequantization processing and fully utilizing the advantages of low bit width operation and small memory occupation of the micro-quantized format in the convolution operation. Here, the micro-quantized format can include the MX quantization format proposed by OCP and the extended MX quantization format proposed by the embodiments of the present disclosure.

[0013] Further, in some embodiments, when the operation number bit width is asymmetric, only the data part of the operation number needs to be processed without processing the scaling factor of the operation number, thereby simplifying the operation. Moreover, such number conversion processing makes it possible to support mixed execution of operations of operation numbers of different data types.

[0014] Optionally or additionally, in some embodiments, the result of the convolution operation, i.e., the output neuron, is also subjected to fine-grained quantization processing. This fine-grained quantization processing facilitates hardware online quantization and supports multiple data types of the quantized scaling coefficients, thereby improving the algorithm accuracy.

[0015] Optionally or additionally, in some embodiments, the scaling coefficients obtained through the extended MX quantization can also be compressed, thereby reducing the waste of storage space and bandwidth. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present embodiments will become more apparent from the following detailed description read in conjunction with the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation in which like reference numerals represent similar, equivalent or corresponding elements, features and aspects. In the drawings:

[0017] FIG. 1 shows a structural diagram of a board card according to an embodiment of the present disclosure;

[0018] FIG. 2 shows a structural diagram of a combined processing device according to an embodiment of the present disclosure;

[0019] FIG. 3 shows a schematic diagram of the internal structure of a processing core when the computing device is a single-core or multi-core device;

[0020] FIG. 4 shows a schematic diagram of an MX quantization data format;

[0021] FIG. 5 shows a schematic block diagram of a data quantization device according to some embodiments of the present disclosure;

[0022] FIG. 6 shows a schematic structural block diagram of a comparison circuit according to some embodiments of the present disclosure;

[0023] FIG. 7 schematically shows a schematic structural diagram of a data processing device that can implement an embodiment of the present disclosure;

[0024] FIG. 8 shows an exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0026] It should be understood that the terms "first", "second", "third", and "fourth" and the like in the claims, specification, and drawings of this disclosure can be used for distinguishing between similar objects, and are not necessarily used to indicate a specific sequential order. The terms "include", "includes" and "including" used in the specification and claims of this disclosure are intended to be inclusive in a manner similar to the term "comprising" so as to mean "comprising but not limited to", unless otherwise indicated.

[0027] It should also be understood that the terms used in the specification and the claims of this disclosure are not to be interpreted as specific terminology but are intended to be given their broadest interpretation in order to encompass all equivalent structures and structures that are now known or become known in the future. As used in the specification and claims of this disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or", as used in the specification and claims of this disclosure, are intended to mean one or the other and both of the items so conjoined, along with the possibility of additional items in some embodiments. As used in the specification and claims of this disclosure, the term "about" means that the value is within 10% of the value.

[0028] The specific embodiments of the present disclosure will now be described in detail below with reference made to the accompanying drawings.

[0029] First, the technical terms that can be used in the present disclosure are explained.

[0030] Floating point number: A number is represented in the form of V = (-1) sign *mantissa*2 E by IEEE floating point standard. Wherein, sign is a sign bit, 0 represents a positive number, and 1 represents a negative number; E represents a scale factor (or index), which weights the floating point number, and the weight is 2 raised to the power of E (which can be a negative power); mantissa represents the mantissa, which is a binary fraction. The representation of a floating point number in a computer is divided into three fields, and these fields are encoded:

[0031] (1) A single sign bit sign directly encodes the sign s;

[0032] (2) The scale factor field of k bits encodes the scale factor, exp = e(k-1)... e(1) e(0);

[0033] (3) The decimal field mantissa of n bits encodes the mantissa. But the encoding result depends on whether the scale factor field is all 0.

[0034] Fixed-point number: a way of representing numbers in a computer, characterized by a fixed position of the decimal point. Fixed-point numbers can represent integers or decimals, depending on the implied position of the decimal point. Fixed-point numbers can be composed of three parts: a shared exponent, a sign, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers that need to be quantized; the sign indicates the positive or negative of the fixed-point number. The mantissa determines the number of significant digits of the fixed-point number, i.e. the precision.

[0035] Bit width: the number of bits used to represent data.

[0036] Quantization: the process of converting high-precision numbers into lower-precision data that occupies less memory space.

[0037] Dequantization: the process of converting low-precision numbers into higher-precision data.

[0038] FP*: represents a floating-point data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc., and the value of * indicates the bit width of the data represented by this data type.

[0039] BF16, TF32: data types suitable for deep learning, which sacrifice precision and range to improve computing performance.

[0040] int*: represents an integer data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc., and the value of * indicates the bit width of the data represented by this data type.

[0041] ExMy: a data format, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, and y represents the mantissa bit width.

[0042] MX: refers to the Microscaling specification (see "OCP Microscaling Formats (MX) Specification Version 1.0").

[0043] Extended MX: refers to a quantization scheme proposed in the embodiments of the present disclosure.

[0044] Microscaling quantization: includes MX and extended MX, as well as other quantizations with similar data structures, and the quantization granularity is determined by the amount of data that can be processed by the operator at a time.

[0045] It should be noted that in the embodiments of the present disclosure, the data that needs to be quantized or has been quantized or dequantized can be various data in the neural network, including but not limited to input neurons, weights, output neurons, gradients, etc. The neural network provided by the embodiments of the present disclosure can be applied to various fields and can process various types of data. For example, in image recognition and processing, face recognition, object detection, image classification, medical image analysis, etc., the data types can include pixel data, image features, etc.; in natural language processing (NLP), language translation, sentiment analysis, text summarization, speech recognition, etc., the data types can include text data, word embeddings, sentence structures, etc.; in speech recognition and processing, intelligent assistants, automatic caption generation, speech-to-text conversion, etc., the data types can include audio signals, spectrograms, etc.; in recommendation systems, personalized content recommendation, product recommendation, ad placement, etc., the data types can include user behavior data, item features, rating data, etc.; in medical health, disease diagnosis, drug discovery, gene sequence analysis, etc., the data types can include medical records, biomarker data, genomic data, etc.; in the financial field, risk assessment, fraud detection, stock market prediction, etc., the data types can include transaction data, user credit scores, market data, etc.; in autonomous driving, vehicle perception, decision making, path planning, etc., the data types can include sensor data, environmental features, traffic signals, etc.; in games and entertainment, game AI, virtual reality, animation generation, etc., the data types can include game state data, user interaction data, etc.; in scientific research, physical simulation, chemical compound prediction, astrophysics analysis, etc., the data types can include experimental data, simulation results, observation data, etc.; generative AI applications, etc. Generative AI refers to artificial intelligence technology that uses complex algorithms, models, and rules to learn from large-scale data sets to create new original content, such as but not limited to the ability to create text, pictures, sounds, videos, and code in multiple types of content. Accordingly, the input neuron data processed can be unquantized or already quantized image data, audio data, video data, speech data, text data, document data, etc., and the corresponding output neurons can also be unquantized or already quantized image data, audio data, video data, speech data, text data, document data, etc. The output of the neural network can include a likelihood score that an image belongs to a certain object category, a likelihood score that a document is about a certain topic, a likelihood score that a text segment in a target language is a correct translation of a text segment in a source language, or a likelihood score that a text segment is a correct transcription of a spoken utterance, etc. These output data can also be quantized or unquantized. Through the quantization / dequantization processing of the data, the computational complexity and the calculation resources can be reduced while ensuring the processing effect of these processing tasks.The data quantization scheme of the embodiments of the present disclosure can also be used in the inference and training of a neural network. By performing the quantization process as described in the embodiments of the present disclosure on the data of the neural network, the accuracy of the task can be ensured when the neural network performs one or more processing tasks, while the speed and performance of the inference and training of the neural network model are improved.

[0046] Exemplary hardware architecture

[0047] FIG. 1 shows a structural schematic diagram of a board card 10 according to an embodiment of the present disclosure. As shown in FIG. 1, the board card 10 includes one or more chips 101, which is a system-level chip, or a system on chip (SoC), integrated with one or more combined processing devices. The combined processing device is an artificial intelligence operation unit used to support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in cloud intelligent fields. A significant feature of cloud intelligent applications is the large amount of input data, which has high requirements for the storage and computing capabilities of the platform. The board card 10 of this embodiment is suitable for cloud intelligent applications and has a large off-chip storage, on-chip storage and powerful computing capability.

[0048] The chip 101 is connected to an external device 103 through an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card or a wifi interface, etc. The data to be processed can be transmitted from the external device 103 to the chip 101 through the external interface device 102. The computing result of the chip 101 can be transmitted back to the external device 103 through the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a peripheral component interconnect express (PCIe) interface, etc.

[0049] The board card 10 further includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected and transmits data with the control device 106 and the chip 101 through a bus. The control device 106 in the board card 10 is configured to regulate the state of the chip 101. For this purpose, in one application scenario, the control device 106 can include a single-chip microcontroller, also known as a microcontroller unit (MCU).

[0050] FIG. 2 is a structural diagram showing the combined processing apparatus in the chip 101 of this embodiment. As shown in FIG. 2, the combined processing apparatus 20 includes a computing apparatus 201, an interface apparatus 202, a processing apparatus 203, and a storage apparatus 204.

[0051] The computing apparatus 201 is configured to perform user-specified operations, mainly implemented as a single-core or multi-core intelligent processor to perform the computation of deep learning or machine learning, which can interact with the processing apparatus 203 through the interface apparatus 202 to jointly complete the user-specified operations.

[0052] The interface apparatus 202 is used to transmit data and control instructions between the computing apparatus 201 and the processing apparatus 203. For example, the computing apparatus 201 can obtain input data from the processing apparatus 203 via the interface apparatus 202 and write it into the storage apparatus on the computing apparatus 201. Further, the computing apparatus 201 can obtain control instructions from the processing apparatus 203 via the interface apparatus 202 and write it into the control buffer on the computing apparatus 201. Alternatively or optionally, the interface apparatus 202 can also read data in the storage apparatus of the computing apparatus 201 and transmit it to the processing apparatus 203.

[0053] The processing apparatus 203 serves as a general-purpose processing apparatus to perform basic controls including but not limited to data transfer, start and / or stop of the computing apparatus 201, etc. Depending on the implementation, the processing apparatus 203 can be one or more types of processors of a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors, including but not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned before, only in terms of the computing apparatus 201 of the present disclosure, it can be considered as having a single-core or homogeneous multi-core structure. However, when the computing apparatus 201 and the processing apparatus 203 are considered together, both of them are considered to form a heterogeneous multi-core structure.

[0054] The storage device 204 is used to store data to be processed, which can be a DRAM (Dynamic Random Access Memory), a DDR (Double Data Rate) memory, usually with a size of 16G or more, for saving data of the computing device 201 and / or the processing device 203.

[0055] FIG. 3 shows a schematic diagram of the internal structure of a processing core of the computing device 201 in FIG. 2 when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, data mining, etc. The computing device 301 includes three major modules: a control module 31 (also referred to as a controller), a computation module 32 (also referred to as a computation unit), and a storage module 33 (also referred to as a memory).

[0056] The control module 31 is used to coordinate and control the work of the computation module 32 and the storage module 33 to complete the task of deep learning, which includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is used to fetch instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding results as control information to the computation module 32 and the storage module 33.

[0057] The computation module 32 includes a vector computation unit 321 and a matrix computation unit 322. The vector computation unit 321 is used to perform vector operations, which can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix computation unit 322 is responsible for the core computation of the deep learning algorithm, i.e., matrix multiplication and convolution.

[0058] The storage module 33 is used to store or transfer related data, including a neuron storage unit (NRAM) 331, a weight storage unit (WRAM) 332, and a direct memory access module (DMA) 333. The NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; the WRAM 332 is used to store the convolution kernel of the deep learning network, i.e., the weight; and the DMA 333 is connected to the DRAM 204 through a bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204. It should be noted that the NRAM and the WRAM here can be two storage areas divided in the logical storage space of the same memory, or two independent memories, which are not specifically limited here.

[0059] Example MX quantized data

[0060] Figure 4 shows a schematic diagram of MX quantization data format, i.e. the components of MX quantization data. As shown, MX quantization data format is characterized by three parts, namely, scaling factor (X, shared scale), scaling block size (K, Scaling Block size), and private element data (P i ) within the block.

[0061] Scaling factor X is related to the data type or encoding method of scaling factor. Private element (P i ) is also related to its data type or encoding method. Scaling block size (K) represents the number of private elements P i within the block. All K elements P i within the block have the same data type, and thus have the same bit width. Scaling factor X is shared among the K elements. The data type of element and the data type of scaling factor can be selected independently. In a sense, MX can be regarded as a mechanism for constructing vector data type from scalar data type. The values V1, …, V k represented in an MX block can be determined as follows: V i = XP i .

[0062] As shown, W represents the number of bits used to encode shared scaling factor X, i.e. the bit width of scaling factor X; d represents the number of bits used to represent each element P i , i.e. the bit width of P i . Therefore, for an MX block containing K elements, it can be encoded by (W+Kd) bits.

[0063] Some common formats of MX and related parameters are given in Table 1. Table 1

[0064] In the above Table 1, ExMy represents the notation of scalar format with x exponent bits and y mantissa bits. For example, FP8(E4M3) represents FP8 format with 1 sign bit, 4 exponent bits and 3 mantissa bits. In the case where y is 0 (e.g. E8M0), the format does not include sign bit.

[0065] Further, in MX quantization, the calculation method of scaling factor X is as follows:

[0066] (1) determine the maximum power of 2 that is less than or equal to the maximum value of absolute values of K elements P i ;

[0067] (2) divide the maximum power of 2 by the power of 2 of the maximum value that can be represented by the data type of element P i , and set the calculation result as scaling factor Scale X.

[0068] The present inventors have found that, for the MX quantization manner, when the original data is processed into a super-low precision data type such as F4 (a data format indicating that the data bit width is 4 bits), the task accuracy of a large model will be sharply reduced, that is, the accuracy of the calculation result of the task will be greatly affected. Taking an image recognition task as an example, the task accuracy or the processing effect of the task can be embodied as the accuracy of the image recognition result, that is, whether the calculation result of the image recognition is accurate. The image quantization data obtained after the foregoing quantization manner will affect the accuracy of the image recognition result.

[0069] The present inventors have also found that, if the data type of the scaling factor used in the quantization manner is the same as that of the original data, although the above-mentioned accuracy reduction problem can be solved to some extent, such a manner will result in greater resource overhead (for example, storage space, area overhead required for hardware support, etc.).

[0070] An exemplary extended MX quantization method

[0071] Based on the above findings, the present disclosure provides a brand-new data quantization scheme, in which a scaling factor is determined based on the maximum absolute value of the data to be quantized, and a scaling coefficient is obtained by converting the scaling factor in data type. Since the converted scaling coefficient has a non-zero mantissa, the accuracy of the quantized data is improved, so that the processing task performed based on the quantized data after quantization according to the embodiments of the present disclosure has good accuracy. In addition, the granularity of quantization is set based on the single processing capacity of hardware, so that the quantization of one granularity block can be completed by a single processing of hardware, thereby making it more convenient to implement quantization / dequantization online by hardware, facilitating pipelining design, and enabling a neural network model using such hardware as a computing carrier to implement quantization operation in each stage of inference and training, thereby improving the operation efficiency of the neural network model.

[0072] In the extended MX quantization scheme, the calculation method of the scaling coefficient is as follows:

[0073] (1) determining the maximum absolute value in a group of data to be quantized;

[0074] (2) obtaining the maximum value that can be represented by the target data type of the quantized data to be quantized;

[0075] (3) determining the scaling factor corresponding to the group of data to be quantized based on the ratio between the maximum absolute value and the maximum value;

[0076] (4) converting the scaling factor from a first data type to a second data type to obtain a scaling coefficient, wherein the bit width of the second data type can be smaller than that of the first data type.

[0077] The target data type is the data type of the quantized data obtained after the quantization of the data to be quantized. Before the quantization process, the target data type can be set as needed, and the precision thereof is generally lower than that of the original data type of the data to be quantized. In some embodiments, the original data type of the data to be quantized can include at least one of, for example, FP16, BF16, FP32, FP64, TF32, and the like; and the target data type can include at least one of, for example, FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, and the like. Among them, the floating-point data types such as FP8, FP6, FP4, and the like can support multiple data formats. Taking FP4 as an example, according to the rule of "sign bit + exponent bit + mantissa bit", there can be multiple divisions, and usually the sign bit occupies 1 bit, and the common forms of the combination of the exponent bit and the mantissa bit are E3M0 and E2M1. Similarly, FP8 can support data formats such as E4M3 and E5M2, and FP6 can support data formats such as E3M2 and E2M3.

[0078] Due to the bit width limitation of each data type, each data type has a maximum value that it can represent. For example, when the target data type is FP4, the maximum value that it can represent is 6; and when the target data type is int8, the maximum value that it can represent is 127. Generally, the maximum value has a one-to-one correspondence with the target data type, and can be obtained by, for example, table lookup and the like.

[0079] Since the data type of the absolute maximum value is the same as the original data type of the data to be quantized, the data type of the scaling factor obtained by the ratio (i.e., the first data type) is the same as the original data type of the data to be quantized, or the bit width of the first data type can be greater than the bit width of the original data type.

[0080] The first data type is the data type of the scaling factor, and the second data type is the data type of the scaling coefficient. Through the data type conversion, the bit width of the second data type is smaller than that of the first data type, so that the precision of the scaling coefficient is smaller than that of the scaling factor. In some embodiments, the conversion of the scaling factor to the scaling coefficient can be realized by a conversion instruction, such as cvt_to_scale_dtype(). In other embodiments, the conversion of the scaling factor to the scaling coefficient supports multiple round modes, such as rn (round to nearest, ties to even).

[0081] In some embodiments, the second data type can be represented as ExMy, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, y represents the mantissa bit width, and x+y < the bit width of the original data type. For example, assuming the original data type is FP16, then x+y < 16. The specific values of x and y can be set as needed or according to the hardware environment, x is a positive integer, and y > 0. In some embodiments, 3 < x+y < the bit width of the original data type. In some embodiments, 8 < x+y < the bit width of the original data type. In some embodiments, the second data type can include at least one of, for example, E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, E3M9, and the like. In some preferred embodiments, y > 0. In further preferred embodiments, the second data type can be E8M2 or E8M3.

[0082] For ease of understanding, still taking the original data type as FP16 as an example, when 8 < x+y < the bit width of the original data type, the bit width represented by the second data type ExMy of the scaling coefficient (i.e., x+y) can be between 8 bits and 16 bits. Using such a scaling coefficient to quantize the operation data of the neural network, and using the quantized data obtained after quantization to perform inference and training of the neural network, etc., the accuracy of the obtained neural network is between 8-bit representation (e.g., E8M0 represented in MX) and original 16-bit representation (e.g., E5M10, E8M7, i.e., FP16, BF16). Compared with the MX format, such a scaling coefficient of the present disclosure is advantageous to improve the accuracy of the quantized data; compared with the case where the data type of the scaling coefficient is the same as the original data type, such a scaling coefficient of the present disclosure is advantageous to reduce the hardware resource overhead or storage overhead, etc.

[0083] Further, since the mantissa bit width affects the accuracy of quantization, a larger mantissa bit width can provide a scaling coefficient with higher accuracy, thereby preserving more original data details in the quantization process, and thus, compared with the E8M0 format scaling coefficient in MX quantization, when the mantissa bit width y of the scaling coefficient in the present disclosure is greater than 0, it is advantageous to improve the accuracy of the quantized data and the accuracy of the processing task.

[0084] In addition, the embodiments of the present disclosure can make the scaling factor support multiple data types by converting the scaling factor, so as to more flexibly adapt to different data characteristics and accuracy requirements. Regarding the selection of the exponent bit width and the mantissa bit width of the scaling factor, the distribution characteristics of the to-be-quantized data can be considered. The greater the exponent bit width of the scaling factor, the more values can be represented. If the numerical range of a group of to-be-quantized data varies greatly, the exponent bit width of the scaling factor can be set to be greater; if the numerical range of a group of to-be-quantized data varies less, the exponent bit width of the scaling factor can be set to be smaller, and the mantissa bit width can be set to be greater.

[0085] For example, it is assumed that the second data type needs to be set to 8 bits. If the absolute value numerical range of a group of to-be-quantized data involves 2 -127 ~ 2 127 , the second data type can be set to E8M0; if the absolute value numerical range of a group of to-be-quantized data involves 2 -4 ~ 2 4 , the second data type can be set to E3M5. Of course, it can be understood that the bit width of the second data type can not be limited to 8 bits, but can also be set to 5 bits or 4 bits, etc. according to needs.

[0086] After the scaling factor is determined in the above manner, the scaling factor can be used to quantize the to-be-quantized data corresponding to the scaling factor, or can be used to dequantize the quantized data.

[0087] In some embodiments, the quantization processing can include: dividing the to-be-quantized data by the scaling factor to obtain a quantization intermediate result; and converting the quantization intermediate result to obtain quantized data having a target data type. The data type conversion can also be implemented by a number conversion instruction, such as cvt_to_scale_dtype(). The number conversion instruction can be a software and hardware instruction at each level, such as a microinstruction inside a processor, or an ISA instruction, or a function instruction implemented by a high-level language at each level on the software side, etc.

[0088] The dequantization processing is generally a process of restoring low-precision data to high-precision data. In some embodiments, the dequantization processing of the quantized data by using the scaling factor can include: multiplying the quantized data by the scaling factor. In other embodiments, the quantized data can be converted from the target data type to a required data type, and then multiplied by the scaling factor, wherein the accuracy of the required data type is higher than that of the target data type. In yet other embodiments, the dequantization processing can restore the quantized data to the to-be-quantized data before the quantization operation.

[0089] As can be seen from the second data type of the scaling coefficients converted from the front, the bit width of a single scaling coefficient is 3-15 bits, for example, E8M4 requires 12 bits, and generally 2 bytes (2B) of storage space need to be allocated, thus certainly not full. This inevitably leads to waste of storage space and transmission bandwidth. In order to solve this problem, in some embodiments, two or more scaling coefficients can be compressed, thereby saving storage space and bandwidth.

[0090] In some embodiments, after two or more groups of to-be-quantized data are quantized to obtain two or more scaling coefficients by using the foregoing method, the two or more scaling coefficients can be compressed. Then, the corresponding groups of to-be-quantized data are quantized using the compressed scaling coefficients respectively, to output quantized data having a target data type.

[0091] Specifically, a compression processing step can be included after the scaling coefficients are obtained by using the foregoing method, and the compression processing step specifically includes: determining a shared exponent of Z scaling coefficients to be compressed, Z≥2; determining Z new mantissas corresponding to the Z scaling coefficients; and bundling and outputting the shared exponent and the Z new mantissas for associated use. That is, after compression, the exponents of the Z scaling coefficients are multiplexed, and the respective mantissas are different, thereby at least Z-1 exponent bit widths can be saved.

[0092] In some implementations, determining the shared exponent of the Z scaling coefficients to be compressed can include: setting the maximum exponent in the Z scaling coefficients as the shared exponent. Selecting the maximum exponent as the shared exponent can ensure the precision of the number with a larger value as much as possible, because generally the number with a larger value has a greater impact on the calculation result.

[0093] In some implementations, determining the Z new mantissas corresponding to the Z scaling coefficients can include: setting the new mantissa of the scaling coefficient with an exponent less than the maximum exponent as 0; and setting the new mantissa of the scaling coefficient with an exponent equal to the maximum exponent as the original mantissa. It can be understood that, for the scaling coefficient with an exponent equal to the maximum exponent, since the exponent is unchanged, the mantissa part is also unchanged; and for the scaling coefficient with an exponent less than the maximum exponent, after the exponent is enlarged to the maximum exponent, if the equivalence is to be maintained, the mantissa will be proportionally reduced, which exceeds the numerical range that the mantissa part can represent. Therefore, the new mantissa of the scaling coefficient is set to 0. This setting will cause a certain precision loss, but relative to the storage and bandwidth benefits, the precision loss is acceptable in some cases.

[0094] In some implementations, the shared exponent and the Z new mantissas can be bundled and output in sequence for associated use. For example, the shared exponent is first, followed by the new mantissa of the first scaling coefficient, the new mantissa of the second scaling coefficient, and so on, until the new mantissa of the Zth scaling coefficient.

[0095] Then, the corresponding group of to-be-quantized data is quantized using the compressed scaling factor to output quantized data of a target data type. Since the exponent and mantissa part of the scaling factor with an exponent less than the maximum exponent will change after compression, in order to maintain consistency between quantization and dequantization, the corresponding group of to-be-quantized data is quantized using the compressed scaling factor.

[0096] In some implementations, the second data type of the Z scaling factors to be compressed is the same. For example, the Z scaling factors are all of the E8M4 type. In this way, the shared exponent bit width of the compressed multiplexing is 8 bits, and the Z new mantissas are 4Z bits.

[0097] In some implementations, the Z scaling factors belong to adjacent Z groups of to-be-quantized data. In this way, the data can be processed sequentially without the need for additional information to indicate the scaling factors compressed together.

[0098] In some implementations, Z can be 2 or 4 or 8 or other. Those skilled in the art can select a suitable Z value according to the number K of to-be-quantized data in a group of to-be-quantized data. For example, when K is small, Z can be larger; when K is large, Z needs to be smaller, thereby ensuring that the benefit of using the compression scheme is greater than the benefit of multiplying K by Z.

[0099] Similarly, when dequantizing the extended MX quantized data using the compressed format, the scaling factor needs to be decompressed first. Specifically, in response to the scaling factor being in the compressed format, the compressed scaling factor is decompressed during dequantization; and the decompressed scaling factor is used to dequantize the quantized data.

[0100] According to the compression method and bundling method described above, the bundled Z scaling factors can be decoded into Z independent scaling factors. Specifically, the shared exponent can be combined with the Z new mantissas to obtain Z decompressed scaling factors.

[0101] It can be understood that in some embodiments, in addition to calculating and saving the scaling factor, the reciprocal of the scaling factor is also calculated and saved to facilitate converting the division operation between the to-be-quantized data and the scaling factor into a multiplication operation between the to-be-quantized data and the reciprocal of the scaling factor during quantization. The reciprocal of the scaling factor has the same data type as the scaling factor. Therefore, the compression and decompression processing described above can also be applied to the reciprocal of the scaling factor, which will not be described here.

[0102] In the extended MX quantization scheme, the scaling factor is determined based on the maximum absolute value of the data to be quantized, which can ensure the accuracy of the quantized data with the maximum absolute value. Since in the quantization process, the greater the value of the data to be quantized, the greater the loss of accuracy, which will have a greater impact on the task calculation result, therefore, by ensuring the accuracy of the data with the maximum absolute value, it can help to reduce the loss of task accuracy, thereby compared with the way of calculating the scaling factor in MX, the extended MX quantization scheme can help to ensure the task accuracy.

[0103] In addition, compared with the quantization method in which the data type of the scaling factor is the same as the original data type, in the disclosed embodiments, the scaling factor with the second data type is obtained by the number of revolutions, which can help to reduce the storage space required for storing the scaling factor and the hardware overhead required for performing the task, while ensuring the task accuracy.

[0104] In some embodiments, the number of data to be quantized in the group of data to be quantized can be determined according to the single processing amount and / or the number of input ports of the processing circuit performing the quantization processing; or the number of data to be quantized in the group of data to be quantized can be determined according to the single operation amount and / or the number of input ports of the operation unit using the quantized data for operation.

[0105] For example, the number of data to be quantized in the group of data to be quantized can be determined according to the single processing amount (i.e., the amount of data that can be processed in a single run) of the processing circuit performing the quantization processing, or according to the single operation amount (i.e., the amount of data that can be operated in a single run) of the operation unit using the quantized data for operation such as multiply-accumulate operation or vector inner product operation, and grouping is performed according to such granularity. According to such setting, a group of data to be quantized sharing a scaling factor can be processed at a time within one operation period, without the need to span multiple operation periods to process the data to be quantized sharing a scaling factor, thereby facilitating hardware implementation and making full use of hardware computing power.

[0106] For another example, the number of data to be quantized in the group of data to be quantized can be determined according to the number of input ports of the processing circuit performing the quantization processing or the number of input ports of the operation unit using the quantized data for operation, and grouping is performed according to such granularity. According to such setting, a group of data to be quantized can be input at a time using the number of input ports of the hardware, so that a group of data to be quantized sharing a scaling factor can be processed at a time in one input, without the need to input multiple times to process a group of data to be quantized, thereby facilitating hardware implementation, reducing hardware processing waiting time, and reducing IO calling overhead.

[0107] For example, assuming that a single operation of the processing circuit or the operator can process 32 data, and the corresponding number of input ports can also be 32, then every 32 pieces of to-be-quantized data can be divided into a group. Assuming that a single operation of the processing circuit or the operator can process 64 data, and the corresponding number of input ports is less than 64, for example, the number of input ports is 32, then the to-be-quantized data can be grouped according to the number of input ports, so that the number of to-be-quantized data in each group is equal to the number of input ports. At this time, 32 data are taken as a group for one-time input, and two groups of data can be operated in parallel. Alternatively, 64 pieces of to-be-quantized data can still be divided into a group, and two-time input is implemented to realize one-time operation of one group of to-be-quantized data. Assuming that a single operation of the processing circuit or the operator can process 32 data, and the corresponding number of input ports is more than 32, then 32 pieces of to-be-quantized data can be selected to be divided into a group to realize one-time input of one group of to-be-quantized data and one-time processing.

[0108] In some embodiments, the to-be-quantized data can be elements of tensor data, and dividing the plurality of to-be-quantized data into a plurality of groups can include: dividing the plurality of to-be-quantized data of the tensor data into a plurality of groups in at least one dimension of the tensor data. For example, assuming that the tensor data is A(B, S), B and S represent the sizes of the first dimension and the second dimension respectively, if the division is performed in the second dimension, and the grouping granularity is taken as 32 (that is, each group contains 32 pieces of to-be-quantized data), then 32 consecutive pieces of to-be-quantized data in the S dimension are divided into a group, (B×S / 32) groups are obtained, and each group is taken as a data block. After grouping, the shape (that is, the dimensions and sizes of the tensor) of the tensor composed of the data blocks can be represented as (B, S / 32), that is, the size of the second dimension after grouping is S / 32.

[0109] For another example, assuming that the tensor data is A'(B, S, D), B, S, and D represent the sizes of the first dimension, the second dimension, and the third dimension respectively, if the division is performed in the third dimension, and the grouping granularity is taken as 64, then 64 consecutive pieces of to-be-quantized data in the third dimension are divided into a group, (B×S×D / 64) groups are obtained, and each group is taken as a data block. After grouping, the shape of the tensor composed of the data blocks can be represented as (B, S, D / 64), that is, the size of the third dimension after grouping is D / 64.

[0110] It can be understood that dividing the plurality of to-be-quantized data into a plurality of groups can not be limited to dividing in only one dimension of the tensor data, but can also be divided in multiple dimensions as needed. The grouping granularity can not be limited to the aforementioned 32 or 64, but can be set as needed, for example, can be 16, 128, etc. The grouping granularity is related to the operation capability or input port limitation of the processing circuit / operator described above, and will not be described here again.

[0111] It needs to be further explained that, as the number of groups of packets (i.e. the number of data blocks) increases, the number of scaling coefficients also increases accordingly, and the number of data that the scaling coefficients can represent is more, which is beneficial to further improve the data quantization precision. In particular, as the number of scaling coefficients increases, the total bit width of the exponent bit width and the mantissa bit width increases, so it can represent more data than the number of data represented by the E8M0 data format in MX quantization, and further compared with MX quantization, the data precision that the scaling coefficients can represent in the extended MX quantization of the embodiments of the present disclosure will be higher.

[0112] Therefore, on the basis of the technical scheme of obtaining the scaling coefficient based on the maximum absolute value and through data type conversion, the quantization granularity is increased by grouping quantization, and corresponding scaling coefficients are provided for each group of to-be-quantized data, which is beneficial to further improve the data precision after quantization and the task precision.

[0113] An exemplary extended MX quantization apparatus

[0114] FIG. 5 shows a schematic block diagram of an apparatus for data quantization according to some embodiments of the present disclosure. As shown in the figure, the apparatus 500 can include a first processing circuit 510 and a second processing circuit 520, wherein the first processing circuit 510 can be configured to output scaling coefficients, and the second processing circuit 520 can be configured to perform quantization processing. Specifically, the first processing circuit 510 can be configured to: determine a maximum absolute value in a group of to-be-quantized data; obtain a maximum value that can be represented by a target data type of the to-be-quantized data after quantization; determine a scaling factor corresponding to the group of to-be-quantized data based on a ratio between the maximum absolute value and the maximum value; and convert the scaling factor from a first data type to a second data type to output the scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type. The second processing circuit 520 can be configured to perform quantization processing on the group of to-be-quantized data using the scaling coefficient to output quantized data having the target data type.

[0115] In some embodiments, the first processing circuit 510 can be further configured to: determine the number of to-be-quantized data in the group of to-be-quantized data according to the single processing amount and / or the number of input ports of the second processing circuit 520; or determine the number of to-be-quantized data in the group of to-be-quantized data according to the single operation amount and / or the number of input ports of an operator that uses the quantized data for operation.

[0116] In some embodiments, the apparatus 500 can further include a compression circuit 540. The compression circuit 540 is configured to compress two or more scaling coefficients after quantization of two or more groups of to-be-quantized data. At this time, the second processing circuit 520 is configured to perform quantization processing on the corresponding group of to-be-quantized data using the compressed scaling coefficient, respectively, to output quantized data having the target data type.

[0117] Correspondingly, when the device 500 needs to dequantize the extended MX quantized data using the compressed format, the scaling factor needs to be decompressed first. In these embodiments, the device 500 can further include a decompression circuit (not shown in the figure). The decompression circuit is configured to decompress the compressed scaling factor in response to the scaling factor being in the compressed format. At this time, the circuit (e.g., the second processing circuit 520 or the third processing circuit 530) in the device 500 that performs the dequantization process can be configured to dequantize the quantized data using the decompressed scaling factor.

[0118] The operations performed by the above first processing circuit 510, second processing circuit 520, compression circuit 540, and decompression circuit have been described in detail in the foregoing in conjunction with the extended MX quantization method, and will not be described again here.

[0119] In some embodiments, the first processing circuit 510 can include a comparison circuit 511, which can be configured to determine the maximum absolute value in the set of data to be quantized; a first division circuit 512, which can be configured to obtain the maximum value that can be represented by the target data type after quantization of the data to be quantized, and determine the scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and a first conversion circuit 513, which can be configured to convert the scaling factor from the first data type to the second data type to output the scaling factor. In some embodiments, the first division circuit 512 can obtain the maximum value by looking up a table, etc.

[0120] In some embodiments, the second processing circuit 520 can include a second division circuit 521 and a second conversion circuit 522, wherein the second division circuit 521 can be configured to perform division operation on the data to be quantized and the scaling factor, and the second conversion circuit 522 can be configured to perform conversion operation on the operation result (i.e., intermediate quantization result) of the division operation to output the result after quantization of the data to be quantized (i.e., quantized data).

[0121] In other embodiments, the second processing circuit 520 can be further configured to dequantize the quantized data using the scaling factor. For example, the second processing circuit 520 can further include a multiplication circuit configured to perform multiplication operation on the quantized data and the scaling factor. In yet other embodiments, the second conversion circuit 522 can be further configured to perform conversion operation on the quantized data to convert the quantized data from the target data type to the required data type for the dequantization process; and the multiplication circuit can be configured to perform multiplication operation on the converted quantized data and the scaling factor to obtain the dequantization result. In the case where the required data type is the original data type of the data to be quantized, the dequantization process can restore the quantized data to the data to be quantized before the quantization operation.

[0122] It can be understood that the dequantization processing can not be limited to the foregoing implementation by the second processing circuit 520, but can also be implemented by other circuits. For example, in some other embodiments, the apparatus 500 can further include a third processing circuit 530 (shown in a dashed box) for dequantizing the quantized data by using the scaling factor. The third processing circuit 530 can receive the quantized data output by the second processing circuit 520, for example, and can receive the scaling factor output from the first processing circuit 510. In some embodiments, the third processing circuit 530 and the second processing circuit 520 can further include a storage circuit, or the second processing circuit 520 includes a storage circuit, for storing the quantized data, so that the third processing circuit 530 can read the quantized data from the storage circuit for dequantization processing.

[0123] In some other embodiments, when one piece of data to be quantized is input to the comparison circuit 511, the comparison circuit 511 can output the absolute value of the one piece of data to be quantized as the maximum absolute value. In some other embodiments, when multiple pieces of data to be quantized are input to the comparison circuit 511, the comparison circuit 511 can output the maximum absolute value of the multiple pieces of data to be quantized. In some other embodiments, the comparison circuit 511 can be further configured to divide the multiple pieces of data to be quantized into multiple groups and output the maximum absolute value of each group of data to be quantized; the first division circuit 512 can be further configured to determine a scaling factor corresponding to each group of data to be quantized; and the first conversion circuit 513 can be further configured to determine a scaling factor corresponding to each group of data to be quantized based on each scaling factor. In some embodiments, the data to be quantized is an element of tensor data, and the comparison circuit 511 can be further configured to divide the multiple pieces of data to be quantized of the tensor data into multiple groups in at least one dimension of the tensor data.

[0124] It can be understood that the above description is exemplary and not limiting. For example, the first processing circuit for obtaining the scaling factor can not be limited to the implementation by the comparison circuit, the first division circuit and the first conversion circuit as described above, but can also be implemented by, for example, a lookup table (circuit) or the like. Since the dividend (the maximum value that can be represented by the target data type) is a constant, the lookup table or the like can be used to directly determine the scaling factor according to the input data, to implement the functions of the above division operation and data type conversion. In addition, in order to facilitate understanding of the grouping function implemented by the comparison circuit, the following will be exemplarily described in conjunction with FIG. 6.

[0125] FIG. 6 shows a schematic structural block diagram of a comparison circuit according to some embodiments of the present disclosure. As shown in the figure, the comparison circuit 511 can include a comparator tree composed of multiple levels of comparators, for example, can include a layer a, a layer b, a layer c, …, a layer z, each level including one or more comparators, the number of comparators in each level decreasing in turn, wherein the outputs of the multiple comparators in the previous level of the adjacent level can be input into the comparators in the next level. In actual applications, the outputs of the required levels can be selected as needed by software or hardware to achieve the grouping of multiple to-be-quantized data and obtain the absolute value maximum of each group of to-be-quantized data. For ease of understanding, further examples will be given in conjunction with FIG. 6.

[0126] As further shown in FIG. 6, the layer a includes a comparator a1, a comparator a2, a comparator a3, …, a comparator an, where n represents the number of comparators in the layer a; the layer b includes a comparator b1, a comparator b2, etc.; the layer c includes a comparator c1, etc.; and the layer z includes a comparator z1. It can be understood that the comparator tree shown in the figure is exemplary, and the number of levels of the comparator tree can be more or less as needed, and the number of comparators included in each level can also be set as needed.

[0127] In some embodiments, the number of comparators n in the layer a can be the same as the number of input to-be-quantized data, so that each to-be-quantized data is input into a corresponding comparator in the layer a, and the output can be gradually reduced in turn as the number of comparators in the subsequent levels decreases, to obtain the required grouping result and the absolute value maximum of each group. For example, assuming that the input layer receives 128 to-be-quantized data, and the layer a is provided with 128 comparators, then each to-be-quantized data is input into a corresponding comparator in the layer a. At this time, the output of the layer a can achieve the division of the 128 to-be-quantized data into 128 groups. Assuming that the layer b includes 64 comparators, then the output of each two comparators in the layer a can be input into a comparator in the next level, i.e., the layer b, for example, the output of the comparator a1 and the comparator a2 in the layer a can be input into the comparator b1 in the layer b. At this time, the layer b outputs 64 absolute value maximums, and at the same time, achieves the division of the 128 to-be-quantized data into 64 groups. Assuming that the layer c includes 32 comparators, then the output of each two comparators in the layer b can be input into a comparator in the next level, i.e., the layer c, for example, the output of the comparator b1 and the comparator b2 in the layer b can be input into the comparator c1 in the layer c. At this time, the layer c outputs 32 absolute value maximums, and at the same time, achieves the division of the 128 to-be-quantized data into 32 groups. In turn, until the last level, i.e., the layer z, outputs one absolute value maximum, at this time, the division of the 128 to-be-quantized data into 1 group is achieved.

[0128] Still taking the embodiment as an example, in actual application, assuming that it is required to divide each 4 data in 128 to-be-quantized data into a group, and a total of 32 groups are divided, the output of the c layer can be selected as the output of the comparison circuit 411 through, for example, programming, logical control, or a selector, so that the first division circuit can determine the scaling factor corresponding to each group of to-be-quantized data in the 32 groups of to-be-quantized data based on the 32 outputs (i.e., 32 absolute maximum values) of the c layer. Therefore, the setting of the comparator tree provides a grouping manner with multiple granularities, and the user can flexibly select the output of a certain level in the comparator tree according to the required quantization granularity.

[0129] In addition, it needs to be noted that the structure of the comparator tree can not be limited to the output of two comparators in each level corresponding to the input of one comparator in the next level, but can also be set according to requirements, for example, the output of three comparators or a larger number of comparators is input into one comparator in the next level. For example, assuming that the a layer is provided with 128 comparators, and the b layer is provided with 4 comparators, that is, the output of 32 comparators in the a layer is input into one comparator in the b layer, the output of the b layer can divide the 128 to-be-quantized data into 4 groups, and output the absolute maximum value in each group of 32 to-be-quantized data.

[0130] In other embodiments, the number n of comparators in the a layer can be different from the number of to-be-quantized data input, which is illustrated below. Assuming that the number of comparators in the a layer is less than the number of to-be-quantized data, a plurality of to-be-quantized data can be input into one comparator in the a layer, and then the a layer divides a plurality of to-be-quantized data into a group output less than the number of to-be-quantized data. In this embodiment, the to-be-quantized data input into each comparator in the a layer can also be controlled through, for example, address decoders, multiplexers (MUXs), or other logical circuits before the a layer (for example, the input layer).

[0131] For example, assuming that the input layer receives 128 to-be-quantized data, and the a layer is provided with 64 comparators, then each two to-be-quantized data is input into a corresponding comparator in the a layer. At this time, the output of the a layer can divide the 128 to-be-quantized data into 64 groups. The number of comparators in the subsequent levels is halved in turn, and finally one absolute maximum value is output in the last level, at which time the 128 to-be-quantized data are divided into 1 group.

[0132] The above describes the scheme of grouping and outputting the maximum absolute value of the comparison circuit according to the embodiments of the present disclosure by taking FIG. 6 as an example. It can be understood that the structure of the comparison circuit shown in the figure and the foregoing description are exemplary and not limited, for example, the last level of the comparison circuit 511 can not be limited to only including one comparator in the figure, and can include multiple comparators as needed, for example, taking the c layer in the figure as the last level. For another example, the comparator circuit 511 can not be limited to only including one comparator tree, and multiple comparator trees can be set as needed, so that parallel grouping and the like can be implemented. The comparison circuit of the embodiments of the present disclosure can not be limited to being implemented by the foregoing comparator tree, and can also be implemented based on a radix sort and the like.

[0133] It can also be understood that the grouping manner in the embodiments of the present disclosure can not be limited to the comparison circuit, and can also be implemented by combining software. For example, the grouping parameters (including the number K of the quantized data in each group and the like) are set by software, and the comparison circuit can read the continuous K quantized data as a group according to the grouping parameters, and directly output the maximum absolute value in each group of quantized data.

[0134] In the embodiments of the present disclosure, when referring to “micro-granularity quantization format”, “micro-granularity format”, “micro-granularity quantization”, it refers to the data format including three parts shown in FIG. 4, but the specific element data type, scaling block size, scaling coefficient data type and the like are not limited to those listed in Table 1, but can have more types of combinations. In other words, the micro-granularity quantization format can include the MX quantization format proposed by OCP, can include the extended MX quantization format proposed by the embodiments of the present disclosure, and can include any data format conforming to the structure of FIG. 4.

[0135] In the embodiments of the present disclosure, when referring to “instructions”, it can include software instructions, hardware instructions, firmware instructions or any combination thereof. Software instructions generally refer to statements or commands in a programming language, which are high-level and abstract instructions, for example, including function calls, machine codes, byte codes and the like. Hardware instructions, also known as machine instructions or instruction set architecture (ISA) instructions, are low-level commands that can be directly recognized and executed by computer hardware, for example, including processor instructions, CPU instructions and the like. Firmware instructions include operation codes, microcodes and the like. Further, different CPU architectures and different instruction sets can have different hardware instruction sets, which can be divided into CISC (Complex Instruction Set Computer), RISC (Reduced Instruction Set Computer) and VLIW (Very Long Instruction Word) according to the complexity of computer instruction set, hardware design, execution speed, complexity of compiler and instruction format and the like.

[0136] Exemplary convolution operation principle

[0137] The convolutional layer in the neural network model can perform convolution operation, and feature extraction is performed by applying a convolution kernel (also referred to as a weight, a filter, etc.) to the input neurons (also referred to as input data, input feature maps) to perform convolution processing. The convolutional layer can include a plurality of convolution kernels, and each element of the convolution kernel corresponds to a weight coefficient and a bias amount bias.

[0138] In a conventional 3D convolution operation, it is assumed that the input neuron or input feature map tensor shape in the convolutional layer is represented as X[N Hi Wi Ci], which represents N groups of input feature maps, each group containing Hi x Wi x Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also referred to as the number of input channels. The tensor shape of the convolution kernel is represented as K[Co Kh Kw Ci], that is, the convolutional layer has Ci x Co convolution kernels with a size of Kh x Kw, where Ci is the number of input channels, Co is the number of output feature maps (or the number of output channels), and Kh and Kw are the height and width of the convolution kernel, respectively. The output result is Y[N Ho Wo Co], that is, there are N groups of output feature maps, and each group of output feature maps contains Ho x Wo x Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Then, the mathematical calculation formula of the simplified convolution operation can be represented as follows: in,jc,jh,jw =∑0≤ic≤ci,0≤ih≤kh,0≤iw≤kwXin,ic,jh×sh+ih,jw×sw+iw×K jc,ic,ih,iw (1)

[0139] In the above formula, sh and sw are the convolution strides in the height and width directions, and the size of the convolution stride affects the size of the output feature map. The bias, the padding pad, and the dilation are ignored in the formula, and it is assumed that the input data X has been padded and the convolution kernel has been dilated. The formula also ignores the N dimension, and the forward calculation of the neural network model in the N dimension is independent. When the convolution kernel works, it scans the input feature according to a certain stride, performs matrix element multiplication and summation on the input feature in the convolution window, and adds the bias amount.

[0140] Note that in this document, input feature map, input data, neuron or input neuron are used interchangeably; convolution kernel, filter or weight are used interchangeably. In addition, H (height) and Y dimension are used interchangeably, W (width) and X dimension are used interchangeably. Accordingly, the H dimension of input feature map can be denoted as Hi or Yi, the H dimension of output feature map can be denoted as Ho or Yo, and the W dimension is similar. In embodiments of the present disclosure, each convolution output point has a corresponding convolution window, and the shape of the convolution window is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the result of the multiplication and accumulation of the input feature map and the weight in its convolution window.

[0141] Exemplary data processing apparatus

[0142] In embodiments of the present disclosure, a convolution operation scheme compatible with MX quantization format is proposed. In this convolution operation, any one or more of input neurons, weights and output neurons can be in a fine-grained quantized data format. When any one or both of input neurons and weights are in a fine-grained quantized format, the dequantization process of data can be integrated in the convolution operation without the need for special dequantization of data in advance, thereby avoiding separate dequantization processing and making full use of the advantages of low bit width operation and small memory occupation of fine-grained quantized format in convolution operation. Further, when the output neurons or the input neurons and weights that are not fine-grained quantized are required to be in a fine-grained quantized format, they can be quantized online by hardware circuit.

[0143] FIG. 7 exemplarily shows a schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure. It can be understood that this structure can be regarded as a refinement of the internal structure of the single processing core in FIG. 3, or as a functional division block diagram combined on the basis of multiple processing cores shown in FIG. 3. As shown in FIG. 7, the data processing apparatus 700 can include a control circuit 710, a storage circuit 720 and an operation circuit 730, which are connected to each other to transmit various data and instructions.

[0144] The control circuit 710 is responsible for processing various functions on the data processing apparatus 700, including but not limited to control, instruction fetching, decoding, calculation, etc. The control circuit 710 may, for example, include the control module 31 in FIG. 3.

[0145] The storage circuit 720 can be used to store information, which at least includes pre-processing and / or post-processing information, and can also include intermediate information that needs to be cached during processing, which may, for example, be various RAMs or on-chip caches shown in FIG. 3. In some embodiments, the storage circuit 720 can be configured to store input neurons, weights, convolution operation results (output neurons) and / or cached intermediate results.

[0146] The operation circuit 730 can be configured to perform various operation operations according to relevant instructions. For example, the operation circuit 730 can be configured to perform convolution operation on the input neuron and the weight value under the control of the control circuit 710. At this time, the operation circuit 730 can include a convolution circuit 731.

[0147] In some embodiments, the control circuit 710 can be configured to parse the convolution instruction, and control the storage circuit 720 and the operation circuit 730 to perform convolution operation processing on the input neuron and the weight value.

[0148] The convolution instruction indicates that the input neuron and the weight value are convolved to obtain the output neuron. In some embodiments, the convolution instruction can support that any one or more of the input neuron, the weight value and the output neuron is micro-quantized. For example, in different application scenarios, the data formats of the input neuron, the weight value and the output neuron can be different. In the first convolution layer of the neural network, the input neuron can be unquantized, while the weight value can be quantized offline, for example, and the output neuron is required to be quantized before being transmitted to the next convolution layer. At this time, in the convolution operation of the next convolution layer, the output neuron received from the previous layer is quantized as the input neuron, and the weight value is also quantized, and the output neuron obtained by the convolution operation is required to be quantized before being input to the next convolution layer. Those skilled in the art can understand that the above scenarios are only exemplary, and there can be various other application occasions, so that any one or more of the input neuron, the weight value and the output neuron is micro-quantized.

[0149] From the above description of the micro-quantization format, it can be known that the micro-quantization format includes three parts: the scaling coefficient (X, scale), the granularity block size (K, Scaling Block size), and the private element data (P i data) in the block. Generally, the granularity block size is pre-set, for example, 32. Therefore, two parts of the scaling coefficient X and the element data P i are required to be provided for the micro-quantized parameters in the convolution operation. Compared with the convolution operation performed on the non-quantized format data, the memory access requirement for the scaling coefficient X is increased.

[0150] In some embodiments, the operation circuit 730 is configured to perform convolution operation on the data of the input neuron and the data of the weight value according to the convolution instruction described above, and fuse the scaling coefficient of the input neuron and / or the scaling coefficient of the weight value in the convolution operation. More specifically, the scaling coefficient can be fused in the convolution operation in units of the granularity block size of the micro-quantized.

[0151] In some embodiments, the convolution circuit 731 can be configured to perform the convolution operation as follows: calculate the partial sum of each granularity block in units of granularity block size T quantized at fine granularity; and, accumulate the partial sum of each granularity block within a single convolution window to obtain the output neuron value of the corresponding convolution window. It can be understood that the granularity block size T is usually no more than the hardware single operation amount, and the operation data involved in a convolution window usually includes multiple granularity blocks, so the calculation of the partial sum can be performed in units of granularity blocks, and the partial sum is accumulated.

[0152] Further, in some embodiments, the convolution circuit 731 can be configured to calculate the partial sum of each granularity block as follows: within each granularity block, calculate the product sum of the data of the input neurons and the data of the weights; calculate the product of the product sum and the scaling coefficient of the input neurons and / or the scaling coefficient of the weights to obtain the partial sum of the granularity block. Since the element data in the same granularity block shares the same scaling coefficient, the product sum of the elements can be calculated first, and then multiplied by the common scaling coefficient, so that the dequantization is fused in the convolution operation.

[0153] The above formula (1) gives the convolution operation of non-quantized format data. Depending on which dimension or dimensions of the tensor data the granularity block is divided in, the order of the convolution operation can also be adjusted accordingly. In some embodiments, when the input neurons and the weights are in fine-granularity quantized format, and the input channels Ci dimension has been quantized at fine granularity according to the granularity block size T, formula (1) can be adjusted to formula (2) as follows:

[0154] In formula (2), Y is the output neuron, with Ho, Wo, Co dimensions; A is the input neuron, with Ci, Hi, Wi dimensions; W is the weight, with Ci, Kh, Kw, Co dimensions, K h , K w are the sizes of the Kh and Kw dimensions respectively, the input neurons and / or the weights have been quantized at fine granularity in the Ci dimension, and the granularity block size of the quantization is T, are the scaling coefficients of the nth granularity block of the input neuron A and the weight W respectively, s h , s w are the convolution step sizes of the weights in the Kh and Kw dimensions respectively.

[0155] As can be seen from formula (2), in the above embodiment, since the input neurons and the weight values share a scaling coefficient for quantization every T data in the Ci dimension, and the convolution operation is accumulated in the Ci dimension, the multiplication and accumulation of the input neuron data and the weight data can be performed first to obtain a partial sum in units of granularity block size T, and then multiplied by the respective corresponding scaling coefficient to realize the dequantization and obtain the dequantized partial sum. The dequantized partial sum can be continuously accumulated with the dequantized partial sum of other granularity blocks in the same convolution window to obtain the output point corresponding to the convolution window, that is, the value of the output neuron.

[0156] Specifically, in some embodiments, the convolution circuit 731 can be configured to perform the above convolution operation as follows: in the input channel Ci dimension, the partial sum of each granularity block is calculated in units of granularity block size T; in the input channel Ci dimension, the partial sum of each granularity block is accumulated to obtain a first accumulation result; and in the height H and width W dimensions, each first accumulation result in a single convolution window is cyclically accumulated to obtain the output neuron value corresponding to the convolution window.

[0157] In the above embodiment, the micro-granularity quantization is the sharing of the scaling coefficient for quantization in the Ci dimension according to T data. When the number of data in the Ci dimension does not meet the alignment to T, the remaining data is shared with the same scaling coefficient when split to less than T data.

[0158] It can be understood that although the above formula (2) is given according to the input neurons and the weight values both having scaling coefficients, it is also suitable for the case where either or both of the input neurons and the weight values are non-quantized data. For example, when the input neurons are non-quantized data and the weight values are quantized data, there is no scaling coefficient of the input neurons, and only the multiplication of the product sum and the scaling coefficient of the weight values is needed to obtain the partial sum of the granularity block. In other words, the scaling coefficient of the input neurons at this time is 1, and vice versa. It can be understood that formula (2) is also compatible with the case where both the input neurons and the weight values are non-quantized data, and the scaling coefficients of the input neurons and the weight values are both 1.

[0159] In some application scenarios, there can also be a case where the bit width of the input neuron data and the weight data is not the same. Generally, the operation hardware supports the same bit width of the operands, that is, the symmetric input case. In order to better adapt to the hardware, a preprocessing circuit can be added in the operation circuit 730.

[0160] As shown in FIG. 7, in some embodiments, the operation circuit 730 can further include a first number conversion circuit 732, which can be configured to, in response to the convolution instruction indicating that the data bit width of the input neuron is different from that of the weight, convert the data of the input neuron and / or the data of the weight, so that the data bit widths of the input neuron and the weight are the same. Then, the convolution circuit 731 can perform the aforementioned convolution operation based on the data of the input neuron and the data of the weight having the same data bit width, and the scaling coefficient of the input neuron and / or the scaling coefficient of the weight.

[0161] The convolution instruction of the embodiments of the present disclosure can be compatible with various input cases. The following describes the processing scheme that can be taken for various cases.

[0162] (1) Non-quantized weight and non-quantized input neuron, same bit width. In this case, no number conversion is needed.

[0163] (2) Non-quantized weight and non-quantized input neuron, different bit widths. In this case, the first number conversion circuit 732 can convert the weight and / or the input neuron, so that the data bit widths of the weight and the input neuron are the same. For example, the data having a shorter bit width in the weight and the input neuron can be converted into data having a longer bit width, so as to unify the bit widths. Of course, the weight and the input neuron can also be converted into data having a longer bit width than the original bit widths of the two, thereby improving the precision.

[0164] (3) Micro-granularity quantized weight or input neuron, same bit width as non-quantized input neuron or weight. In this case, there can be two processing schemes. One is not to perform number conversion, but to perform dequantization in the convolution circuit 731 in the process of performing the convolution operation, for example, in the manner described in the foregoing formula (2). The other is to perform dequantization on the micro-granularity quantized weight or input neuron in the first number conversion circuit 732, so that subsequent convolution operation can be performed in the conventional convolution operation manner (for example, formula (1)) together with the non-quantized input neuron or weight.

[0165] (4) The micro-quantized weight or input neuron has different bit width from the non-quantized input neuron or weight. In this case, conversion is needed, but similar to (3), there are two processing schemes. One is that the first conversion circuit 732 converts the weight and / or input neuron so that the data bit width of the weight and input neuron is the same. Generally, the data bit width of the micro-quantized data is shorter, so the micro-quantized weight or input neuron can be converted to the same bit width (longer bit width data) as the non-quantized input neuron or weight. The scaling factor of the micro-quantized data is processed by the convolution circuit in the process of performing convolution operation. Similarly, the weight and input neuron can also be converted to data with bit width longer than the original bit width of the two. The other is that the first conversion circuit performs dequantization while converting the micro-quantized weight or input neuron, so that the subsequent convolution operation can be performed with the non-quantized input neuron or weight in the conventional convolution operation manner.

[0166] (5) The micro-quantized weight and input neuron have the same bit width. In this case, no conversion is needed, and the scaling factor is processed by the convolution circuit 731 in the process of performing convolution operation, for example, in the operation manner described above in connection with formula (2).

[0167] (6) The micro-quantized weight and input neuron have different bit width. In this case, the first conversion circuit 732 can convert the weight and / or input neuron so that the data bit width of the weight and input neuron is the same. For example, the data with shorter bit width among the weight and input neuron can be converted to data with longer bit width, so that the bit width is unified. The scaling factor of the micro-quantized data is processed by the convolution circuit in the process of performing convolution operation. Similarly, the weight and input neuron can also be converted to data with bit width longer than the original bit width of the two. For example, the data type of the input neuron after micro-quantization is FP8, and the data type of the weight after micro-quantization is FP4, and the data type of the two can be converted to FP16, and the convolution operation is performed based on the data type of FP16. Note that the converted FP16 is still micro-quantized data with scaling factor.

[0168] In the above various schemes, compared with the scheme of first dequantizing and then performing convolution, the advantage of the convolution fusion dequantization of the micro-quantized scaling factor in the convolution operation process (i.e., convolution fusion dequantization) of the embodiments of the present disclosure is that the performance acceleration ratio can be improved. For example, in the case that the input neuron and the weight are both micro-quantized data, such as FP8, if the dequantization is performed first, and both are converted to quantized FP16, the data number that can be processed by one stroke is N / 2, and the operation bandwidth is N Byte. However, if the operation circuit is directly inputted with the quantized data in the format of FP8, and the dequantization is performed in the operation process using the scaling factor X, the operation bandwidth N can be fully used, and the computing power is twice that of the previous one.

[0169] It can be understood that, in the case where rotation is required to be performed, the real memory access on the rotation channel needs to access the storage circuit (e.g., the storage circuit 720) in the scale of the channel bandwidth / rotation ratio, so as to avoid the data amount after rotation exceeding the channel bandwidth. For example, if the rotation before rotation is FP4 and the rotation after rotation is FP8, and the bandwidth is 128B, then the data amount of a single access is 128B / (8 / 4) = 64B.

[0170] As mentioned before, compared with the convolution operation performed on the non-quantized format data, the convolution instruction suitable for fine-grained quantization in the present embodiment increases the memory access requirement for the scaling coefficient X.

[0171] In some implementations, the scaling coefficient X can be stored continuously with the k element data P i The scaling coefficient X and the element data P i may be accessed together, for example, through one address. At this time, the convolution instruction can only provide one memory access address for a single parameter.

[0172] In some implementations, the scaling coefficient X can be stored independently with the k element data P i The scaling coefficient X and the element data P i may be accessed independently, for example, through different addresses respectively. At this time, the convolution instruction needs to provide two memory access addresses for a single parameter.

[0173] In some embodiments, the scaling coefficient X can take the foregoing compressed format to save storage space and transmission bandwidth. In this case, the convolution instruction can increase the corresponding indication information to indicate the compressed format of the scaling coefficient X. At this time, the operation circuit 730 can further include a decompression circuit 734 configured to, in response to the convolution instruction indicating that the scaling coefficient therein is in the compressed format, decompress the scaling coefficient in the compressed format to obtain a decompressed scaling coefficient. In some implementations, the compressed format includes a shared exponent of the Z compressed scaling coefficients and Z new mantissas corresponding thereto, where Z≥2. At this time, the decompression circuit 734 is further configured to combine the shared exponent and the Z new mantissas respectively to obtain Z decompressed scaling coefficients for use by the convolution circuit 731.

[0174] In current AI chips, the access to on-chip RAM is usually the bottleneck, and the existing data has already occupied most of the on-chip access bandwidth. For the access demand of the newly added scaling coefficients, two schemes can be used. One scheme is to increase the access bandwidth, for example, to add several ports to the on-chip RAM for the access of the newly added scaling coefficients. The other scheme is to reuse the original ports for the access of the scaling coefficients. For example, a selector is added to the original neuron reading port to select reading neuron data or neuron scaling coefficients; a selector is added to the original weight reading port to select reading weight data or weight scaling coefficients. Similarly, a selector is added to the original output neuron (result) write-back port to select writing back result data or result scaling coefficients.

[0175] In the case of reusing the ports, since time needs to be allocated to read the scaling coefficients, the data cannot be continuously supplied, which will inevitably lead to some performance degradation. Therefore, in some embodiments, the property that the scaling coefficients are shared within the granularity block can be used to fill the bandwidth when reading the scaling coefficients every beat, and the cache is used for subsequent multiple periods.

[0176] Specifically, in some embodiments, the convolution instruction also indicates that the scaling coefficients and their corresponding data reuse the same reading port, and that the scaling coefficients are read every beat, and the corresponding data is read in proportion. It can be understood that in the case of not reusing the port, the reading action between the scaling coefficient reading port and the data reading port can also be controlled in proportion, for example, the scaling coefficient reading port reads one beat of scaling coefficient, and the data reading port reads n beats of corresponding data, so that the scaling coefficient reading port is idle for n-1 beats, which can be used to read other data.

[0177] In these embodiments, as shown in FIG. 7, the data processing apparatus 700 further includes a first cache 741 configured to cache the scaling coefficients read from off-chip to supply the convolution circuit 731 in the operation circuit 730 to perform convolution operation in proportion to the corresponding data.

[0178] The above proportion is determined based on at least the granularity block size T, the data bit width of the scaling coefficient, and the data bit width of the corresponding data. Since one scaling coefficient is shared among K (i.e., T=K, for example, 32) element data, the usage proportion of the two is 1 scaling coefficient to T corresponding data. According to the data bit width X bw of the scaling coefficient and the data bit width P bw of the corresponding data, the usage proportion (calculated in Bytes) of the two can be determined as: 1×X bw :T×P bw .

[0179] For example, when the format of the scaling coefficient is E8M3, the bit width occupies 2 Bytes, the element data is in FP8 format, the bit width occupies 1 Byte, and T = 32, the above ratio is 2:32 = 1:16. When reading in each beat, whether reading the scaling coefficient or reading the corresponding data, the data amount as much as possible to fill the transmission bandwidth is read to fully utilize the transmission performance. For example, assuming that the reading / transmission bandwidth is 128B, then each reading of the scaling coefficient (128B / 2B = 64 scaling coefficients) reads 16 beats of element data (128B x 16 / 1B = 64 x 32). The read scaling coefficient is cached in the first cache. When the operation circuit performs convolution operation, 128B (for example, filling the operation bandwidth) of element data is supplied, and 128B / 16 = 8B of scaling coefficient is correspondingly supplied from the first cache. Thus, the scaling coefficient read in one beat can stay in the first cache for multiple beats, thereby alleviating the bandwidth tension problem.

[0180] It can be understood that when the scaling coefficient adopts the compressed format, the above ratio can be adjusted according to the compression ratio. For example, when the format of the scaling coefficient is E8M3, every 2 adjacent scaling coefficients are compressed, and after compression, 2 Bytes are occupied, the element data is in FP8 format, the bit width occupies 1 Byte, and T = 32, the above ratio is 2:2 x 32 = 1:32. It can be seen that every 2B of scaling coefficient is read, 64 element data can be correspondingly read, and 64B of element data needs to be read. The decompression circuit 734 can provide the scaling coefficient cached in the first cache 741 to the convolution circuit 731 after decompression.

[0181] It can also be understood that the cache space can be set for the scaling coefficient of the input neuron and the scaling coefficient of the weight value respectively. For example, two first caches can be set to cache the scaling coefficient of the input neuron and the scaling coefficient of the weight value respectively, and the data is shifted and read in each first cache according to the usage amount of each operation period. Of course, it can also be set in the same cache, and the present disclosure embodiment is not limited in this respect.

[0182] It can also be understood that when the data exists the number of turns, the above ratio is calculated according to the bit width of the data after the turns.

[0183] Similarly, in the case that the output neurons (convolution operation results) need to be quantized in micro-granularity, the content written back to the storage circuit also includes the result data and the result scaling factor. Similarly, since one result scaling factor is shared among K (e.g., 32) result data, when the result data can fill the transmission bandwidth of the write back, the corresponding result scaling factor is not enough. In order to control the effective use of the transmission bandwidth of the write back path, and to avoid the bottleneck of the write port, a second buffer (second buffer 742 in FIG. 7) can also be set to temporarily store the result scaling factor. When the result scaling factor accumulates to a certain amount, for example, can fill the transmission bandwidth, then perform the write back task, and write these result scaling factors to the storage circuit through a write. Therefore, the output order can be controlled to output n shots of result data first, and then output 1 shot of result scaling factor. Similarly, n is determined according to the granularity block size T, the bit width of the result data, and the bit width of the result scaling factor.

[0184] Since the write back of the output result scaling factor of the operation circuit per shot is stored on the storage circuit in the Co-Wo-Ho order, and the micro-granularity quantization will be split and quantized in the Co dimension, there may be discontinuity in the Co dimension, and there may be jumps. At this time, when the result scaling factor is buffered in the second buffer, it is buffered in a Co-continuous manner with a stride. For the output point of Co discontinuity, a certain stride is skipped. The stride is related to the instruction Co, and its calculation method can be: stride = Co / T*X bw , where Co is the total Co, T is the granularity block size, and X bw is the bit width of the scaling factor. For example, assuming that 64 Co are calculated per shot, the next 64 Co of the next Wo are calculated per shot, the granularity block size T of the quantization is 32, then 2 scaling factors X are quantized per shot, and if the total Co is 256, then the storage stride of the scaling factors of the next 2 shots is stride = 256 / 32*X bw , that is, the stride is calculated according to the difference between the starting storage positions of the two scaling factors.

[0185] When writing back the result scaling factor in the second buffer to the storage circuit, a write back with a mask can be used, where the mask indicates the valid data in the current write back period that needs to be written to the storage circuit. In this way, since the result scaling factor is buffered in a stride, there will be empty parts, and through the mask indication, only the data (i.e., valid data) that needs to be written back to the storage circuit can be written back.

[0186] The micro-granular quantization of the output neurons can be performed in the quantization circuit 733 in FIG. 7. In some embodiments, the quantization circuit 733 can be configured to quantize the results of the convolution operation according to the convolution instruction to generate the result data and the result scaling coefficients in a micro-granular manner. Alternatively or additionally, in some embodiments, the quantization circuit 733 can also be configured to quantize the input neurons and the weights that are not quantized in a micro-granular manner according to the convolution instruction. By micro-granular quantization, the computation of high-bit-width data can be converted to the computation of low-bit-width data, reducing the computation power consumption.

[0187] In some embodiments, the quantization circuit 733 can further include a compression circuit (not shown) configured to compress the two or more scaling coefficients after the quantization of the two or more groups of data to be quantized in response to the convolution instruction indicating that the scaling coefficients are in a compressed format. At this time, the quantization circuit quantizes the corresponding groups of data to be quantized using the compressed scaling coefficients, respectively, to output quantized data having a target data type.

[0188] The quantization circuit 733 can take various implementations. In some embodiments, the specific implementation of the quantization circuit 733 can refer to the description of the extended MX quantization described above, which will not be repeated here.

[0189] The above describes the convolution operation scheme compatible with the micro-granular quantization format in combination with FIG. 7. Those skilled in the art can understand that although the various buffers and operation circuits are shown as separate modules in FIG. 7, the buffers and operation circuits can also be combined into one module according to different configurations. For example, the first buffer can be combined with the convolution circuit, and the second buffer can be combined with the quantization circuit. The disclosure embodiments are not limited in this respect.

[0190] FIG. 8 shows an exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of the disclosure. In this data processing method, the control circuit controls the operation circuit to perform a convolution operation on input neurons and weights.

[0191] More specifically, in step 810, the control circuit parses the convolution instruction, wherein the convolution instruction indicates that the convolution operation is performed on the input neurons and the weights to obtain the output neurons, and either or both of the input neurons and the weights are quantized in a micro-granular manner.

[0192] Then, in step 820, the operation circuit performs the convolution operation on the data of the input neurons and the data of the weights according to the convolution instruction under the control of the control circuit, and fuses the scaling coefficients of the input neurons and / or the scaling coefficients of the weights in the convolution operation.

[0193] In some embodiments, step 820 can further include: sub-step 821, in response to the convolution instruction indicating that the data bit width of the input neuron is different from that of the weight, converting the data of the input neuron and / or the data of the weight so that the data bit widths of the input neuron and the weight are the same; and sub-step 822, performing the convolution operation based on the data of the input neuron and the data of the weight having the same data bit width, and the scaling coefficient of the input neuron and / or the scaling coefficient of the weight. Sub-step 821 can be performed by a first conversion circuit, and sub-step 822 can be performed by a convolution circuit.

[0194] In some embodiments, the conversion in sub-step 821 can be converting the shorter bit width data in the input neuron and the weight to longer bit width data.

[0195] In some embodiments, sub-step 822 can be performed as follows: calculating the partial sum of each granularity block in units of a granularity quantization block size T in a loop; and accumulating the partial sum of each granularity block in a single convolution window to obtain the output neuron value corresponding to the convolution window.

[0196] Specifically, calculating the partial sum of each granularity block can include: in each granularity block, calculating the product sum of the data of the input neuron and the data of the weight; and calculating the product of the product sum and the scaling coefficient of the input neuron and / or the scaling coefficient of the weight to obtain the partial sum of the granularity block.

[0197] In some embodiments, step 820 can further include: sub-step 823, performing micro-granularity quantization on the result of the convolution operation according to the convolution instruction to generate the result data and the result scaling coefficient. Sub-step 823 can be performed by a quantization circuit. The specific quantization method can refer to the description above.

[0198] Optionally or additionally, in some embodiments, the method can further include a preprocessing step, for example, micro-granularity quantization on the input neuron and the weight that is not micro-granularity quantized by the quantization circuit according to the convolution instruction. Through micro-granularity quantization, the calculation of high bit width data can be converted to the calculation of low bit width data, reducing the calculation power consumption.

[0199] Those skilled in the art can understand that the steps described in the method flowchart correspond to the various circuits of the data processing apparatus described above in combination with FIG. 7, and therefore the features described above also apply to the method steps, which are not repeated here.

[0200] The disclosure embodiments also provide a processor including the foregoing data processing apparatus. The disclosure embodiments also provide a chip, which can include the processor of any of the embodiments described above in combination with the accompanying drawings. Further, the disclosure also provides a board card, which can include the foregoing chip.

[0201] According to different application scenarios, the electronic device or apparatus of the present disclosure can include a server, a cloud server, a server cluster, a data processing apparatus, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle record device, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an oil smoke exhaust fan; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument, and / or an electrocardiograph. The electronic device or apparatus of the present disclosure can also be applied to the fields of Internet, Internet of Things, data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment, etc. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal, etc. application scenarios related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with small power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete the unified management, scheduling and collaborative work of end-cloud integration or cloud-edge integration.

[0202] It should be noted that for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art can understand that the schemes of the present disclosure are not limited by the order of the described actions. Therefore, those skilled in the art can understand that some steps can be executed in other orders or simultaneously according to the disclosure or teaching of the present disclosure. Further, those skilled in the art can understand that the described embodiments of the present disclosure can be regarded as optional embodiments, i.e. the actions or modules involved therein are not necessarily essential for the implementation of one or more schemes of the present disclosure. In addition, the description of some embodiments of the present disclosure also has different focuses according to the schemes. Therefore, those skilled in the art can understand that the parts not described in detail in one embodiment of the present disclosure can also refer to the relevant description of other embodiments.

[0203] In terms of specific implementation, based on the disclosure and teaching of the present disclosure, those skilled in the art can understand that the several embodiments disclosed by the present disclosure can also be implemented in other manners not disclosed herein. For example, in terms of the units described in the aforementioned electronic device or apparatus embodiments, the units can be split into more than one unit, or some features of the units can be combined or integrated into another system, or a plurality of units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. In terms of the connection relationship between the units or components, the units or components discussed in the foregoing embodiments can be directly or indirectly coupled to each other, and can be in a position to communicate with each other via an interface, and the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0204] In the present disclosure, the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units. The foregoing components or units can be located in the same place or distributed on multiple network units. In addition, according to actual needs, part or all of the units can be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure can be integrated into one unit or each unit physically exists separately. For example, in some embodiments, the quantization circuit as the post-processing circuit can be integrated in the convolution circuit or independent of the convolution circuit. In other embodiments, the quantization circuit as the preprocessing circuit can be integrated in the rotation circuit or independent of the rotation circuit. The embodiments of the present disclosure are not limited in this regard.

[0205] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, i.e., specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include but is not limited to physical devices, and the physical devices can include but are not limited to transistors or memristors, etc. In view of this, various apparatuses (such as computing apparatuses or other processing apparatuses) described herein can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM (read only memory), and RAM (random access memory), etc.

[0206] The above detailed description of the embodiments of the present disclosure is made, and the principles and implementation manners of the present disclosure are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present disclosure and its core idea; meanwhile, for those skilled in the art, according to the idea of the present disclosure, the specific implementation manners and application ranges will be changed, and the above description of the embodiments should not be understood as a limitation of the present disclosure.

Claims

A data processing apparatus, comprising: A control circuit configured to parse a convolution instruction that instructs an input neuron and weights to perform a convolution operation to obtain an output neuron, wherein either or both of the input neuron and weights are particle-quantized. as well as The computation circuit is configured to perform a convolution operation on the data of the input neuron and the data of the weights according to the convolution instruction, and to fuse the scaling factor of the input neuron and / or the scaling factor of the weights in the convolution operation. The data processing apparatus according to claim 1, wherein, The arithmetic circuit includes: A rotation circuit is configured to rotate the data of the input neuron and / or the data of the weights in response to a convolution instruction indicating that the data bit widths of the input neuron and the weights are different, so that the data bit widths of the input neuron and the weights are the same; and A convolution circuit configured to perform a convolution operation based on data and weights of input neurons having the same data bit width, and the scaling factor of the input neurons and / or the scaling factor of the weights. The data processing apparatus according to claim 2, wherein, The arithmetic circuit also includes: The decompression circuit is configured to decompress the scaling factor in the compressed format in response to the convolution instruction indicating that the scaling factor is in a compressed format, so as to obtain the decompressed scaling factor. The data processing apparatus according to claim 3, wherein, The compression format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. The decompression circuit is further configured to: The shared index is combined with the Z new mantissas to obtain Z decompressed scaling factors for use by the convolution circuit. The data processing apparatus according to any one of claims 2-4, wherein, The convolution circuit is further configured to convert the shorter bit-width data in the input neuron and the weights into longer bit-width data based on the convolution instruction. The data processing apparatus according to any one of claims 1-5, wherein, The computational circuitry is configured to perform convolution operations as follows: Using the particle size T as the unit, the partial sum of each particle size block is calculated iteratively; and By iteratively summing the partial sums of each granularity block within a single convolution window, the output neuron value of the corresponding convolution window is obtained. The data processing apparatus according to claim 6, wherein the arithmetic circuitry is configured to calculate the partial sum of each granularity block as follows: Within each granularity block, calculate the sum of the product of the input neuron's data and the weight data; and The product of the sum and the scaling factor of the input neuron and / or the scaling factor of the weights is calculated to obtain a partial sum of the granularity block. The data processing apparatus according to any one of claims 1-7, wherein the arithmetic circuit is configured to perform a convolution operation according to the following formula (1): Where Y is the output neuron, with dimensions Ho, Wo, and Co; A is the input neuron, with dimensions Ci, Hi, and Wi; W is the weight, with dimensions Ci, Kh, Kw, and Co, and K h K w The dimensions are Kh and Kw, respectively. The input neuron and / or the weights have been quantized in the Ci dimension, and the block size is T. s represents the scaling factor for the nth granularity block of the input neuron A and the weights W, respectively. h s w These are the convolution strides of the weights in the Kh and Kw dimensions, respectively. The data processing apparatus according to any one of claims 6-8, wherein the convolution instruction further indicates a scaling factor for each read frame, and reads data corresponding to multiple frames proportionally, the data processing apparatus further includes: A first buffer is configured to cache scaling factors read from off-chip to be supplied to the computing circuitry in proportion to the corresponding data for performing the convolution operation, wherein the proportion is determined at least based on the granularity block size, the data bit width of the scaling factor, and the data bit width of the corresponding data. The data processing apparatus according to claim 9, wherein, The amount of data read per frame fully utilizes the transmission bandwidth. The data processing apparatus according to any one of claims 1-10, wherein, The arithmetic circuit also includes: A quantization circuit is configured to perform particle quantization on the result of the convolution operation according to the convolution instruction to generate result data and result scaling factor; and / or is configured to perform particle quantization on the input neurons and weights that have not been particle quantized according to the convolution instruction. The data processing apparatus according to claim 11, wherein, The quantization circuit includes: A first processing circuit is configured to: determine the maximum absolute value of a set of data to be quantized in the result of the convolution operation; obtain the maximum value that the target data type after quantization can represent; determine a scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type; and The second processing circuit is configured to quantize the set of data to be quantized using the scaling factor to output quantized data having the target data type. The data processing apparatus according to claim 12, wherein, The first processing circuit includes: A comparator circuit is used to determine the maximum absolute value in a set of data to be quantized; The first division circuit is used to obtain the maximum value that the target data type after quantization can represent, and to determine the scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and The first rotation circuit is used to convert the scaling factor from a first data type to a second data type to output the scaling coefficient. The data processing apparatus according to claim 13, wherein, The first processing circuit is further configured to: determine the number of data to be quantized in the set of data to be quantized based on the single processing volume and / or the number of input ports of the second processing circuit; or The number of data points to be quantized in the set of data points is determined based on the single operation quantity and / or the number of input ports of the arithmetic unit that performs the operation using the quantized data. The data processing apparatus according to any one of claims 12-14, wherein, The result of the convolution operation is divided into multiple groups at least in the Co dimension for particle quantization. The data processing apparatus according to any one of claims 12-15, wherein The first data type is the same as the original data type of the data to be quantized, or the bit width of the first data type is greater than the bit width of the original data type; and The second data type is represented as ExMy, where x + y < the bit width of the primitive data type. E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width. The data processing apparatus according to claim 16, wherein y > 0. The data processing apparatus according to any one of claims 12-17, wherein The original data type of the data to be quantized includes at least one of FP16, BF16, FP32, TF32, and FP64; the second data type includes at least one of E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, and E3M9; and / or The target data type includes at least one of FP12, FP8, FP6, FP4, FP2, int8, int4, int2, and int6, and FP12, FP8, FP6, and FP4 support multiple data formats. The data processing apparatus according to any one of claims 12-18, wherein the quantization circuit further comprises a compression circuit configured for: In response to the convolution instruction indicating that the scaling factors are in a compressed format, two or more scaling factors after quantization of two or more sets of data to be quantized are compressed; and The second processing circuit is configured to quantize the corresponding group of data to be quantized using the compressed scaling factor, so as to output quantized data with the target data type. The data processing apparatus of claim 19, wherein the compression circuit is further configured to: Determine the shared exponent of the Z scaling factors to be compressed, Z≥2; Determine the Z new mantissas corresponding to the Z scaling factors; and The shared index is bundled with the Z new tail numbers and output for association. The data processing apparatus of claim 20, wherein the compression circuit is further configured to: Set the largest index among the Z scaling factors as the shared index. The data processing apparatus of claim 21, wherein the compression circuit is further configured to: Set the new mantissa of the scaling factor whose exponent is less than the maximum exponent to 0; and Set the new mantissa, whose exponent is equal to the scaling factor of the maximum exponent, as the original mantissa. The data processing apparatus according to any one of claims 20-22 further comprises one or more of the following: The second data type of the Z scaling factors to be compressed is the same; The Z scaling factors belong to Z adjacent groups of data to be quantized; Z can be 2, 4, or 8. The data processing apparatus according to any one of claims 11-23 further includes: The second buffer is configured to cache the scaling factor of the result so that it is written back to the storage circuit in proportion to the corresponding result data. The data processing apparatus according to claim 24, wherein, The scaling factor of the result is cached in the second cache in a step-by-step manner according to the Co dimension, and when the scaling factor of the result is written back to the storage circuit, it is a masked write, wherein the mask indicates the valid data that needs to be written to the storage circuit in the content of the current write-back cycle. The data processing apparatus according to any one of claims 1-25, wherein the quantized data and scaling factor of each of the input neuron, the weights and / or the output neuron support independent or joint memory access. The data processing apparatus according to any one of claims 1-26, wherein, When one of the input neuron and the weights is non-quantized data, the computation circuit sets the corresponding scaling factor to 1. A processor comprising the data processing apparatus as described in any one of claims 1-27. A board comprising the processor according to claim 28. A data processing method implemented by a data processing device, the data processing device including a control circuit and a calculation circuit, the method comprising: The control circuit parses the convolution instruction, which instructs to perform a convolution operation on the input neuron and weights to obtain the output neuron, wherein either or both of the input neuron and weights are particle-quantized. as well as The computational circuit performs a convolution operation on the data of the input neuron and the data of the weights according to the convolution instruction, and integrates the scaling factor of the input neuron and / or the scaling factor of the weights into the convolution operation.

Images