Quantization and inverse quantization method in large language model and neural network processor

Abstract

The invention relates to a quantization method, an inverse quantization method and a neural network processor in a large language model. A core (namely a quantization unit) in a neural network processor is arranged to execute quantization processing in a large-scale language model reasoning process, and the quantization unit has a data partitioning function, so that the quantization processing efficiency is greatly improved compared with an existing quantization processing core which is limited by the data size (such as block wise) when receiving data to be quantized. And frequent interaction with a cache is not needed, so that online high-efficiency large-model quantification processing is realized. And performing an inverse quantization process in a large language model inference process by setting a core (i.e., an inverse quantization unit) in a neural network processor, and the inverse quantization unit having a data partitioning function, a matrix multiplication function, and a high precision accumulation function (e.g., multiplying by a corresponding quantization parameter), the inverse quantization processing does not need to frequently carry data in a plurality of cores, and the inverse quantization processing efficiency of a large model is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a quantization and dequantization method for a large language model and a neural network processor. Background Technology

[0002] With the widespread use of large language models (LLMs), the amount and richness of data to be processed during inference (e.g., multimodal LLMs support data processing across multiple modalities) are increasing. Related technologies involve quantization and dequantization during inference in large language models to improve efficiency and reduce resource consumption. However, current quantization and dequantization processes frequently interact with caches, and due to the existence of multiple data precisions, data needs to be moved between multiple GPU cores. For example, the Tensor Core does not support high-precision accumulation for FP8 data types, requiring data to be moved to the CudaCore for high-precision accumulation. These factors lead to low efficiency in quantization and dequantization processing, resulting in low efficiency in the inference process of large language models. Summary of the Invention

[0003] This disclosure provides a quantization and dequantization method for large language models, as well as a neural network processor, to at least improve the processing efficiency of quantization and dequantization in large language model inference. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a data quantization processing method for a large language model is provided, comprising:

[0005] During the inference process of the large language model, in response to the processor's quantization instruction for the target network layer in the large language model, the quantized input data to be quantized is obtained from the cache; the quantized input data is a first-precision floating-point data type; the target network layer is any network layer in the large language model that needs to undergo data quantization processing.

[0006] The quantized input data is divided into blocks according to the preset quantization block to obtain multiple input data blocks;

[0007] Based on the target input data in each input data block, quantization parameter processing is performed to obtain the target quantization parameters corresponding to each of the multiple input data blocks; the target input data satisfies preset data statistical conditions.

[0008] The input data in each input data block is quantized according to the target quantization parameters corresponding to each input data block to obtain the quantization result data corresponding to the quantized input data. The quantization result data includes the quantization result of each input data. The quantization result data is a floating-point data type with second precision, which is lower than the first precision.

[0009] According to a second aspect of the present disclosure, a method for data dequantization processing in a large language model is provided, comprising:

[0010] During the inference process of the large language model, in response to the processor's dequantization instruction for the target network layer in the large language model, the processor retrieves the first input data and the second input data to be dequantized from the cache, and retrieves the first quantization parameter and the first quantization block information corresponding to the first input data, as well as the second quantization parameter and the second quantization block information corresponding to the second input data from the cache; the first input data and the second input data are the quantization result data after data quantization processing during the inference process, and both the first input data and the second input data are second-precision floating-point data types;

[0011] The first input data is divided into blocks based on the first quantization block information to obtain multiple first data blocks; and the second input data is divided into blocks based on the second quantization block information to obtain multiple second data blocks.

[0012] Based on the first quantization block information and the second quantization block information, matrix multiplication and dequantization are performed on the plurality of first data blocks and the plurality of second data blocks to obtain dequantization result data; the dequantization result data is a floating-point data type with a first precision; the first precision is higher than the second precision.

[0013] According to a third aspect of the present disclosure, a quantization unit in a neural network processor is provided, the quantization unit comprising:

[0014] The quantization input data acquisition module is used to, during the inference process of the large language model, in response to the processor's quantization instruction for the target network layer in the large language model, acquire the quantization input data to be quantized from the cache; the quantization input data is a first-precision floating-point data type; the target network layer is any network layer in the large language model that needs to undergo data quantization processing.

[0015] The first data segmentation module is used to segment the quantized input data into multiple input data blocks according to the preset quantization block.

[0016] The quantization parameter acquisition module is used to perform quantization parameter processing based on the target input data in each input data block to obtain the target quantization parameters corresponding to each of the multiple input data blocks; the target input data satisfies preset data statistical conditions.

[0017] The quantization processing module is used to perform data quantization processing on the input data in each input data block according to the target quantization parameters corresponding to each input data block, so as to obtain the quantization result data corresponding to the quantized input data. The quantization result data includes the quantization result of each input data. The quantization result data is a floating-point data type with second precision, which is lower than the first precision.

[0018] According to a fourth aspect of the present disclosure, a dequantization unit in a neural network processor is provided, the dequantization unit comprising:

[0019] The data acquisition module is used to, during the inference process of the large language model, respond to the processor's dequantization instruction for the target network layer in the large language model, acquire first input data and second input data to be dequantized from the cache, and acquire first quantization parameters and first quantization block information corresponding to the first input data, and second quantization parameters and second quantization block information corresponding to the second input data from the cache; the first input data and the second input data are quantization result data after data quantization processing during the inference process, and both the first input data and the second input data are second-precision floating-point data types;

[0020] The second data segmentation module is used to segment the first input data based on the first quantization segmentation information to obtain multiple first data blocks; and to segment the second input data based on the second quantization segmentation information to obtain multiple second data blocks;

[0021] The dequantization processing module is used to perform matrix multiplication and dequantization processing on the plurality of first data blocks and the plurality of second data blocks according to the first quantization block information and the second quantization block information to obtain dequantization result data; the dequantization result data is a floating-point data type with a first precision; the first precision is higher than the second precision.

[0022] According to a fifth aspect of the present disclosure, a neural network processor is provided, the neural network processor including a quantization unit as described in the third aspect above, and an inverse quantization unit as described in the fourth aspect above.

[0023] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a method as described in any one of the first aspects above or a method as described in any one of the second aspects above.

[0024] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described in the first aspect of the present disclosure or any of the methods described in the second aspect above.

[0025] According to an eighth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, cause a computer to perform the method described in any one of the first aspects of the present disclosure or the method described in any one of the second aspects above.

[0026] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0027] By setting up a core (i.e., a quantization unit) in a neural network processor to perform quantization processing in the inference process of large language models, and this quantization unit has the function of data block division, compared with the existing cores for processing quantization which are limited by the data size (e.g., block-wise) when receiving the data to be quantized, the quantization unit in this application can receive the quantization input data to be quantized at one time, and then achieve fine-grained quantization processing by dividing the quantization input data into blocks, without frequent interaction with the cache, thereby realizing online and efficient quantization processing of large models.

[0028] Furthermore, by setting up a core (i.e., dequantization unit) in the neural network processor to perform dequantization processing during the inference process of large language models, and this dequantization unit has data block function, matrix multiplication function, and high-precision accumulation function (e.g., multiplying by the corresponding quantization parameter), the dequantization processing does not need to frequently interact with the cache, nor does it need to frequently move data between multiple cores, which greatly reduces memory access and improves the dequantization processing efficiency of large models.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0031] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment.

[0032] Figure 2 This is a flowchart illustrating a data quantization processing method in a large language model according to an exemplary embodiment.

[0033] Figure 3 This is a schematic diagram illustrating an architecture for data quantization processing based on a quantization unit in a neural network processor, according to an exemplary embodiment.

[0034] Figure 4 This is a schematic diagram illustrating the structure of a quantization unit in a neural network processor and the process of performing data quantization processing, according to an exemplary embodiment.

[0035] Figure 5 This is a schematic diagram illustrating a data quantization process according to an exemplary embodiment.

[0036] Figure 6 This is a schematic diagram illustrating a method for data quantization processing in a large language model according to an exemplary embodiment.

[0037] Figure 7 This is a schematic diagram illustrating an architecture for data quantization processing based on an inverse quantization unit in a neural network processor, according to an exemplary embodiment.

[0038] Figure 8 This is a flowchart illustrating a data dequantization processing method in a large language model according to an exemplary embodiment.

[0039] Figure 9 This is a schematic diagram illustrating the structure of a dequantization unit in a neural network processor and the process of performing data dequantization processing, according to an exemplary embodiment.

[0040] Figure 10 This is a schematic diagram illustrating a data dequantization process according to an exemplary embodiment.

[0041] Figure 11 This is a schematic diagram illustrating a method for data dequantization processing in a large language model according to an exemplary embodiment.

[0042] Figure 12 This is a block diagram illustrating an electronic device for data quantization or dequantization in a large language model, according to an exemplary embodiment. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0044] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0045] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include a neural network processor and a central processing unit (CPU).

[0046] In an alternative embodiment, the CPU can be used to send quantization or dequantization instructions to trigger the neural network processor to perform data quantization or dequantization.

[0047] In an optional embodiment, the neural network processor may include a quantization unit and an inverse quantization unit, wherein the quantization unit may have data block functionality; the inverse quantization unit may have matrix multiplication and multiplication by quantization parameters functionality, that is, the inverse quantization unit has both matrix multiplication and multiplication by quantization parameters functionality, performing matrix multiplication and multiplication by quantization parameters in one core, so that the inverse quantization process does not need to frequently move data due to different processing precision.

[0048] In addition, it should be noted that, Figure 1 The illustration shows only one application environment of the quantization and dequantization methods and neural network processors in the large language model provided in this disclosure.

[0049] In the embodiments described in this specification, the CPU and neural network processor can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this connection.

[0050] It should be noted that the sequence of steps shown in the diagram is one possible order and is not strictly required to be followed. Some steps can be performed in parallel without interdependence. The user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to data used for display, training data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0051] Before introducing the method embodiments provided in this application, a brief introduction will be given on the application scenarios, related terms or nouns that may be involved in the method embodiments of this application, so as to facilitate the understanding of those skilled in the art.

[0052] Neural Processing Unit (NPU) is a processor specifically designed for artificial intelligence (AI) applications. It is a hardware chip designed to accelerate artificial intelligence and deep learning tasks.

[0053] CPU: Central Processing Unit.

[0054] SRAM: Static Random-Access Memory, a type of cache.

[0055] LLM: Large Language Model. The large language model in the embodiments of this specification can support unimodal or multimodal input.

[0056] Figure 2 This is a flowchart illustrating a data quantization processing method in a large language model according to an exemplary embodiment. This data quantization processing method can be applied to a quantization unit in a neural network processor, which can be considered a core (e.g., a quantization core) for data quantization processing. Figure 2 As shown, the steps may include:

[0057] In step S201, during the inference process of the large language model, in response to the processor's quantization instruction for the target network layer in the large language model, the quantized input data to be quantized is retrieved from the cache.

[0058] In the embodiments described in this specification, a large language model may include a unimodal or multimodal large language model, and this application does not limit this. The reasoning process of a large language model can refer to the process of using a trained large language model to predict or make decisions on new input data, where prediction or decision mainly refers to quantization and dequantization.

[0059] For example, a large language model can be a pre-trained large model used to implement preset functions, such as text generation, intelligent dialogue, image description generation, and visual question answering, etc., which are not limited in this application. Accordingly, the processing efficiency of text generation, intelligent dialogue, image description generation, and visual question answering can be improved.

[0060] In the embodiments described in this specification, the processor may refer to a central processing unit (CPU), for example... Figure 3 As shown.

[0061] In one example, the quantized input data can be a first-precision floating-point data type. The target network layer can be any network layer in a large language model that requires data quantization; that is, any network layer in an LLM can use this data quantization method when data quantization is required. For example, the first-precision floating-point data type can include 32-bit floating-point numbers (FP32) or 16-bit floating-point numbers (FP16 / BF16). As an example, the type of quantized input data can be image data, such as quantized input image data.

[0062] In one possible implementation, refer to Figure 3 The CPU can send instructions (such as quantization instructions) to the neural network processor, which in turn can invoke the quantization unit to respond to the quantization instruction, retrieving the quantized input data from the SRAM cache, for example... Figure 3 The data is read from the SRAM. Then, data quantization processing can be performed, and the quantization result can be stored in SRAM, for example... Figure 3 The image shows the writing of quantization results (e.g., quantization result data) into SRAM.

[0063] For example, the quantization instruction may carry a quantization data identifier of the quantization input data to be quantized, so that the quantization input data to be quantized can be obtained from the SRAM based on the quantization data identifier. This application does not limit this.

[0064] In step S203, the quantized input data is divided into blocks according to the preset quantization blocks to obtain multiple input data blocks.

[0065] In the embodiments of this specification, in order to enable the quantization unit to receive the quantized data to be quantized at once, the data is divided into blocks after reception to match the quantization granularity. For example, this quantization granularity can be represented by a preset quantization block. Based on this, it can be determined whether the size of the quantized input data is greater than the preset quantization block. If it is, the quantized input data can be divided into blocks according to the preset quantization block to obtain multiple input data blocks (e.g., multiple input image data blocks). The size of each input data block can be the preset quantization block, for example, 1*128.

[0066] In step S205, quantization parameter processing is performed based on the target input data in each input data block to obtain the target quantization parameters corresponding to each of the multiple input data blocks.

[0067] In the embodiments described in this specification, the target input data satisfies preset data statistical conditions, which may include, but are not limited to, the maximum absolute value. This application does not limit this.

[0068] In one possible implementation, the target input data can be divided by the data range corresponding to a second-precision floating-point data type to obtain the target quantization parameters for each of the multiple input data blocks, for example... Figure 4 The quantization parameter "scale" is shown. For example, a second-precision floating-point data type can include an 8-bit floating-point number, with 4 exponent bits and 3 mantissa bits, i.e., one type of FP8 E4M3. Correspondingly, the data range can be 448, for example... Figure 4 As shown, the target quantization parameter scale for each input data block can be obtained by dividing the size of the target input data in each input data block by 448.

[0069] In an optional implementation, the aforementioned preset data statistical condition can be the maximum absolute value; correspondingly, step S205 may include: determining the input data with the largest absolute value in each input data block as the target input data in each input data block; obtaining the data range corresponding to the second-precision floating-point data type; for example, the second-precision floating-point data type is FP8, and the data range corresponding to FP8 is 448. Further, the target quantization parameters corresponding to each of the multiple input data blocks can be determined based on the target input data and data range in each input data block. For example, the target quantization parameter scale corresponding to each input data block can be obtained by dividing the size of the target input data in each input data block by 448. By dividing the quantized input data into blocks and calculating the target quantization parameters separately, the calculation efficiency of the quantization parameters is high.

[0070] In step S207, the input data in each input data block is quantized according to the target quantization parameters corresponding to each input data block to obtain the quantization result data corresponding to the quantized input data.

[0071] In the examples provided in this specification, the quantization result data may include the quantization results of each input data; the quantization result data may be a floating-point data type with a second precision, which may be lower than the first precision to reduce the magnitude of subsequent data processing. For instance, if the quantization input data is quantized input image data, the quantization result data may be quantized image result data.

[0072] As an example, first-precision floating-point data types can include 32-bit floating-point numbers (FP32) or 16-bit floating-point numbers (FP16, BF16); second-precision floating-point data types can include 8-bit floating-point numbers, for example... Figure 5 As shown, FP32 / BF16 / FP16 are quantized input data processed to obtain FP8 type quantized result data, and quantization parameters can be obtained. The type of these quantization parameters can be FP32, and this application does not limit this. It can achieve efficient quantization processing of various high-precision data to low-bit data.

[0073] In one possible implementation, the above-mentioned data quantization processing of the input data in each input data block according to the target quantization parameter corresponding to each input data block to obtain the quantization result data corresponding to the quantized input data may include: dividing the input data in each input data block by the target quantization parameter corresponding to each input data block to obtain the quantization result corresponding to each input data. For example Figure 4 As shown, each input data point is divided by its scale, and the quantization result of each input data point is FP8, which is the quantized result with data type FP8. Based on the quantization results corresponding to each input data point, the quantization result data corresponding to the quantized input data is obtained. In one example, the quantization results corresponding to each input data point can be combined together as the quantization result data. Optionally, the method may further include: transferring the target quantization parameters corresponding to each of the multiple input data blocks and the quantization result data corresponding to the quantized input data to the cache for storage. The quantization parameters and quantization results are stored in SRAM for subsequent inverse quantization.

[0074] In an optional implementation, taking an image as the type of quantized input data (i.e., the quantized input data can be quantized input image data), the format of each input data in the quantized input data can include CHW, HWC, etc., which are not limited in this disclosure. Here, C refers to the number of input channels, H refers to the height of the input image, and W refers to the width of the input image. Quantization granularity refers to block-wise, and the block size can be set to 32, 64, 128, 256, 512, etc. The output can include the number of output channels, the height of the output image, the width of the output image, and the quantization parameter width W / block-wise, such as... Figure 5 As shown. For example, the storage method can use CHW_BLK, where CHW represents the storage order: first, storage along the channel direction, then along the H direction, and finally along the W direction. Each block of stored data contains 16*16 points. This method allows for parallel computation along the channel direction, improving data retrieval efficiency.

[0075] By setting up a core (i.e., a quantization unit) in a neural network processor to perform quantization processing in the inference process of large language models, and this quantization unit has the function of data block division, compared with the existing cores for processing quantization which are limited by the data size (e.g., block-wise) when receiving the data to be quantized, the quantization unit in this application can receive the quantization input data to be quantized at one time, and then achieve fine-grained quantization processing by dividing the quantization input data into blocks, without frequent interaction with the cache, thereby realizing online and efficient quantization processing of large models.

[0076] Reference Figure 4 In one possible implementation, the quantization unit may include modules for instruction decoding, data reading, data writing, data segmentation, quantization parameter calculation, and quantization calculation. However, the data reading module (quantization input data acquisition module) and the data writing module are not included in this implementation. Figure 4As shown in the diagram. The instruction decoding module is used to parse the quantization instructions, and the data reading module can read the quantization input parameters from the SRAM according to the parsed instructions. The data block module (i.e., the first data block module) can be used for block processing and to store multiple input data blocks using the data cache space. The quantization parameter calculation module (quantization parameter acquisition module) can be used to calculate the target quantization parameters and pass them to the quantization calculation module (quantization processing module). In this way, the quantization calculation module can obtain the input data from each input data block from the data cache space, divide it by the target quantization parameter corresponding to each input data block, and obtain the quantization result corresponding to each input data. The data writing module can be used to write the quantization result data into the SRAM. The modules of the quantization unit can be implemented by dedicated hardware circuits or by general-purpose processing hardware (such as field-programmable logic devices FPGA, ARM processors, etc.) combined with executable logic instructions; this application does not limit this implementation.

[0077] Optionally, the quantization unit may include a data cache space, and the method may further include: saving multiple input data blocks to the data cache space. Based on this, the above-described data quantization processing of the input data in each input data block according to the target quantization parameters corresponding to each input data block to obtain the quantization result data corresponding to the quantized input data may include: in response to obtaining the target quantization parameters corresponding to the target input data block, extracting the input data in the target input data block from the data cache space; the target input data block is any one of multiple input data blocks. Further, data quantization processing of the input data in the target input data block according to the target quantization parameters corresponding to the target input data block to obtain the quantization result data corresponding to the quantized input data. By setting the quantization unit to include a block-based cache space for storing multiple input data blocks, it is possible to effectively support quantization data processing larger than block-wise and improve quantization efficiency.

[0078] Furthermore, the method may also include: transferring the target quantization parameters corresponding to each of the multiple input data blocks and the quantization result data corresponding to the quantized input data to a cache for storage. The quantization parameters and quantization results are stored in SRAM for convenient and efficient use during subsequent inverse quantization.

[0079] In one optional implementation, the quantization input data can be labeled with a quantization data identifier to distinguish the quantization result data corresponding to different quantization input data. This quantization data identifier, along with the quantization result data, is stored in SRAM for easy data retrieval and extraction during subsequent inverse quantization. Alternatively, corresponding quantization block information can be associated with the quantization result data, and this quantization block information is stored in SRAM along with the quantization result data.

[0080] For example, the quantization block information can be as follows:

[0081] Quantization result data A (FP8-A), quantization block information A: block A1 (A1-1~A1-10), block A2 (A2-1~A2-10), ...

[0082] Quantization result data B (FP8-B), quantization block information B: block B1 (B1-1~B1-10), block B2 (B2-1~B2-10), ...

[0083] Here, A1-1 to A1-10 and A2-1 to A2-10 can be the quantization results corresponding to different input data in quantized input data A; B1-1 to B1-10 and B2-1 to B2-10 can be the quantization results corresponding to different input data in quantized input data B. Blocks A1 and A2 correspond to the blocks (input data blocks) of the corresponding quantized input data A during the quantization process. Blocks B1 and B2 correspond to the blocks (input data blocks) of the corresponding quantized input data B during the quantization process.

[0084] For example, refer to Figure 6 Multiple input data blocks can include block_0, block_1, and block_2. Block_0 contains input data from 1 to block_0, block_1 contains input data from block_0+1 to block_1, and block_2 contains input data from block_1+1 to block_2. This allows us to determine the three largest absolute values ​​among the three input data blocks: AbsMax_0, AbsMax_1, and AbsMax_2. We then obtain the target quantization parameters for each of the three input data blocks: scale_0, scale_1, and scale_2. Finally, we can obtain the quantization result of each input data block after quantization. Figure 6 The data at the bottom. Accordingly, the quantization result data includes the quantization results for each input data point, ranging from 1 to Block_2.

[0085] For example, suppose the number of input data points in the quantized input data is 100, i.e., 100 input data points. After data block processing, it is divided into 10 groups, i.e., 10 input data blocks. Thus, a target input data point with the largest absolute value can be found in each group, for a total of 10 target input data points. These 10 groups of target input data can be divided by 448 to obtain 10 target quantization parameters. Then, the 10 input data points of the first group are divided by the corresponding target quantization parameter of the first group to obtain the 10 FP8 values ​​of the first group's output. This process is repeated 10 times to obtain 10 groups of output FP8 values. These 10 groups of output FP8 values ​​and their corresponding 10 target quantization parameters can then be stored in SRAM. For example, using the above method of associating quantization block information with quantization result data and storing them together in SRAM, the 10 groups of output FP8 values ​​and their corresponding 10 target quantization parameters can be as follows:

[0086] Data identifier for quantification result data A: Quantification result data A;

[0087] The data identifiers for the quantification result data A are: block A1 (A1-1~A1-10), block A2 (A2-1~A2-10)..., block A10 (A10-1~A10-10).

[0088] The data identifiers for the quantization result data A are: block A1 / scale1, block A2 / scale2, ..., block A10 / scale10.

[0089] Reference Figure 7 and Figure 8 Dequantization processing can be performed by a dequantization unit in a neural network processor. For example, the dequantization unit can parse dequantization instructions and, based on the parsed instructions, read the first input data to be dequantized from SRAM (e.g., ...). Figure 8 The quantization result data A) and the second input data (e.g. Figure 8 The quantization result data (B) is processed by matrix multiplication and multiplication by quantization parameters on the first and second input data to be dequantized. The dequantization result can also be written back to SRAM.

[0090] Figure 9 This is a flowchart illustrating a data dequantization processing method in a large language model according to an exemplary embodiment. This data dequantization processing method can be applied to a dequantization unit in a neural network processor. This dequantization unit can be considered a core (e.g., a dequantization core) in the neural network processor, used to support matrix multiplication calculations and high-precision accumulation processing of matrix multiplication followed by multiplication by quantization parameters. Figure 9 As shown, the method may include the following steps:

[0091] In step S901, during the inference process of the large language model, in response to the processor's dequantization instruction for the target network layer in the large language model, the first input data and the second input data to be dequantized are obtained from the cache, and the first quantization parameter and the first quantization block information corresponding to the first input data, as well as the second quantization parameter and the second quantization block information corresponding to the second input data are obtained from the cache.

[0092] In the embodiments described in this specification, the first input data and the second input data are quantization result data after data quantization processing during the inference process. Both the first input data and the second input data can be floating-point data types with second precision. For example, when the quantization result data is quantized image result data, the first input data and the second input data can be first input image data and second input image data.

[0093] The first quantization parameter corresponding to the first input data is the quantization parameter corresponding to the first input data in the quantization process. Each input data block processed in the quantization process has a corresponding quantization parameter. Based on this, the first quantization parameter corresponding to the first input data may include the target quantization parameter (which may be called the first target quantization parameter) corresponding to each of the multiple quantization blocks in the first input data. The first quantization block information corresponding to the first input data is the data block information of the first input data in the quantization process, that is, the block information of the quantization results included in the first input data.

[0094] Accordingly, the second quantization parameter corresponding to the second input data is the quantization parameter corresponding to the second input data in the quantization process. The second quantization parameter corresponding to the second input data may include the target quantization parameter (which may be called the second target quantization parameter) corresponding to each of the multiple quantization blocks in the second input data. The second quantization block information corresponding to the second input data is the data block information of the second input data in the quantization process, that is, the block information of the quantization results included in the second input data.

[0095] Taking quantized result data A as the first input data as an example, assuming that quantized result data A is divided into 10 input data blocks during the quantization process, the corresponding quantized result data A includes 10 sets of quantization results, and there are 10 first quantization parameters: block A1 / scale1, block A2 / scale2, ..., block A10 / scale10. The first quantization block information can include: block A1 (A1-1~A1-10), block A2 (A2-1~A2-10)..., block A10 (A10-1~A10-10).

[0096] For example, in response to a processor's dequantization instruction for a target network layer in a large language model, the dequantization instruction may carry dequantization data identifiers (e.g., input1, input2). This allows the processor to retrieve the first and second input data to be dequantized, matching the dequantization data identifiers, from the cache. It also retrieves the first quantization parameters and first quantization block information corresponding to the first input data, and the second quantization parameters and second quantization block information corresponding to the second input data. Taking the first input data as an example, the processor retrieves the first quantization parameters corresponding to the first input data from the cache: input1: block A1 / scale1, block A2 / scale2, ..., block A10 / scale10; the retrieved first quantization block information may be: input1: block A1 (A1-1~A1-10), block A2 (A2-1~A2-10)..., block A10 (A10-1~A10-10).

[0097] In step S903, the first input data is divided into blocks based on the first quantization block information to obtain multiple first data blocks; and the second input data is divided into blocks based on the second quantization block information to obtain multiple second data blocks.

[0098] In one example, the first input data can be segmented according to the segmentation method used during the quantization process. For instance, if the first input data is quantized result data A, and the first quantization segmentation information corresponding to quantized result data A includes: segment A1 (A1-1 to A1-10), segment A2 (A2-1 to A2-10)..., segment A10 (A10-1 to A10-10). Multiple first input data can be obtained by directly segmenting the data into groups of 10. Using the same method, multiple second data blocks can be obtained. For example, the first data block and the second data block can be a first image data block and a second image data block.

[0099] In step S905, matrix multiplication and dequantization are performed on multiple first data blocks and multiple second data blocks according to the first quantization parameter and the second quantization parameter to obtain the dequantized result data.

[0100] In the embodiments of this specification, the dequantization result data can be a floating-point data type with a first precision; the first precision is higher than the second precision. For example, based on the first quantization parameter and the second quantization parameter, each first data block and each second data block can be matrix-multiplied to obtain multiple intermediate matrix multiplication results. Then, each intermediate matrix multiplication result is multiplied by the corresponding first quantization parameter and the second quantization parameter to obtain the dequantization result data. For example, the first quantization parameter may include the first target quantization parameter corresponding to each first data block in the first input data, and the second quantization parameter may include the second target quantization parameter corresponding to each second data block in the second input data. Thus, each intermediate matrix multiplication result is multiplied by the corresponding first target quantization parameter and the second target quantization parameter to obtain the data block dequantization result. Further, all obtained data block dequantization results can be added together or combined together as the dequantization result data.

[0101] By setting up a core (i.e., the dequantization unit) in the neural network processor to perform dequantization processing in the inference process of large language models, and this dequantization unit has data block function, matrix multiplication function, and high-precision accumulation function (e.g., multiplying by the corresponding quantization parameter), the dequantization processing does not need to frequently interact with the cache, nor does it need to frequently move data between multiple cores, which greatly reduces memory access and improves the dequantization processing efficiency of large models.

[0102] In one possible implementation, the matrix multiplication and dequantization processes performed on the plurality of first data blocks and the plurality of second data blocks based on the first quantization parameter and the second quantization parameter to obtain dequantized result data include:

[0103] Based on the first quantization parameter, a first target quantization parameter corresponding to each of the plurality of first data blocks is determined; and based on the second quantization parameter, a second target quantization parameter corresponding to each of the plurality of second data blocks is determined.

[0104] Combine any of the first data blocks with any of the second data blocks to determine multiple data block combinations, each data block combination including one of the first data blocks and one of the second data blocks;

[0105] Based on the first target quantization parameter corresponding to the first target data block and the second target quantization parameter corresponding to the second target data block in the target data block combination, matrix multiplication and dequantization are performed on the first target data block and the second target data block to obtain the data block dequantization result; the target data block combination is any of the data block combinations; the first target data block and the second target data block refer to the first data block and the second data block in the target data block combination.

[0106] The dequantized result data is obtained based on the dequantization results of multiple data blocks. For example, the dequantization results of multiple data blocks can be combined together as the dequantization result data.

[0107] In one possible implementation, based on the first target quantization parameter corresponding to the first target data block and the second target data block corresponding to the second target data block in the target data block combination, matrix multiplication and dequantization are performed on the first target data block and the second target data block to obtain the data block dequantization result. This includes: performing matrix multiplication on the first target data block and the second target data block to obtain an intermediate matrix multiplication result; the intermediate matrix multiplication result is a floating-point data type of the second precision; and performing dequantization on the intermediate matrix multiplication result based on the first target quantization parameter and the second target quantization parameter to obtain the data block dequantization result. For example, the intermediate matrix multiplication result can be multiplied by the first target quantization parameter and the second target quantization parameter to obtain the data block dequantization result, for example... Figure 10 and Figure 11 As shown.

[0108] In an application example, consider the quantized result data A obtained from the quantized input data A, and the quantized result data B obtained from the quantized input data B. See below:

[0109] Quantization result data A: Quantization block information A: Block A1 (A1-1~A1-10), Block A2 (A2-1~A2-10)..., Block A10 (A10-1~A10-10). First quantization parameter: scale1 corresponding to block A1, scale2 corresponding to block A2,..., scale10 corresponding to block A10;

[0110] Quantization result data B: Quantization block information B: Block B1 (B1-1~B1-10), Block B2 (B2-1~B2-10)..., Block B10 (B10-1~B10-10). Second quantization parameters: Scale1 corresponding to block B1, Scale2 corresponding to block B2,..., Scale10 corresponding to block B10.

[0111] Since matrix multiplication involves repeatedly multiplying rows by columns, when multiplying a row by a column, each row can include multiple blocks, such as 10 blocks: A1 to A10. Each column can also include multiple blocks, such as 10 blocks: B1 to B10.

[0112] Assuming the target data block combination is (block A1, block B1), we can calculate: (A1-1*B1-1+A1-2*B1-2+…+A1-10*B1-10)*scale1*Scale1 = (block A1, block B1) corresponding to the data block dequantization result. Based on this, we can obtain the data block dequantization result corresponding to any target data block combination, and thus use the data block dequantization results corresponding to all target data block combinations as the dequantization result data. For example, we can add the data block dequantization results corresponding to all target data block combinations together as the dequantization result data. The addition between blocks in each target data block combination is a high-precision accumulation, which can be implemented in one core without moving to other cores for accumulation, resulting in higher efficiency.

[0113] In one possible implementation, embodiments of this specification also provide a quantization unit in a neural network processor, the quantization unit comprising:

[0114] The quantization input data acquisition module is used to, during the inference process of the large language model, in response to the processor's quantization instruction for the target network layer in the large language model, acquire the quantization input data to be quantized from the cache; the quantization input data is a first-precision floating-point data type; the target network layer is any network layer in the large language model that needs to undergo data quantization processing.

[0115] The first data segmentation module is used to segment the quantized input data into multiple input data blocks according to the preset quantization block.

[0116] The quantization parameter acquisition module is used to perform quantization parameter processing based on the target input data in each input data block to obtain the target quantization parameters corresponding to each of the multiple input data blocks; the target input data satisfies preset data statistical conditions.

[0117] The quantization processing module is used to perform data quantization processing on the input data in each input data block according to the target quantization parameters corresponding to each input data block, so as to obtain the quantization result data corresponding to the quantized input data. The quantization result data includes the quantization result of each input data. The quantization result data is a floating-point data type with second precision, which is lower than the first precision.

[0118] By setting up a core (i.e., a quantization unit) in a neural network processor to perform quantization processing in the inference process of large language models, and this quantization unit has the function of data block division, compared with the existing cores for processing quantization which are limited by the data size (e.g., block-wise) when receiving the data to be quantized, the quantization unit in this application can receive the quantization input data to be quantized at one time, and then achieve fine-grained quantization processing by dividing the quantization input data into blocks, without frequent interaction with the cache, thereby realizing online and efficient quantization processing of large models.

[0119] In one possible implementation, the preset data statistical condition is the maximum absolute value; the quantization parameter acquisition module includes:

[0120] The determination submodule is used to determine the input data with the largest absolute value in each input data block, which is then used as the target input data in each input data block.

[0121] The data range acquisition submodule is used to acquire the data range corresponding to the second precision floating-point data type;

[0122] The quantization parameter acquisition submodule is used to determine the target quantization parameters corresponding to each of the multiple input data blocks based on the target input data in each input data block and the data range.

[0123] In one possible implementation, the quantization processing module includes:

[0124] The quantization result acquisition submodule is used to divide the input data in each input data block by the target quantization parameter corresponding to each input data block to obtain the quantization result corresponding to each input data.

[0125] The quantization processing submodule is used to obtain the quantization result data corresponding to the quantized input data based on the quantization result corresponding to each input data.

[0126] In one possible implementation, the quantization unit includes a data cache space, and the device further includes:

[0127] A caching module is used to save the plurality of input data blocks to the data cache space;

[0128] The quantization processing module is further configured to, in response to obtaining the target quantization parameter corresponding to the target input data block, extract the input data from the target input data block from the data cache space; the target data input block is any one of the plurality of input data blocks; and perform data quantization processing on the input data in the target input data block according to the target quantization parameter corresponding to the target input data block to obtain the quantization result data corresponding to the quantized input data.

[0129] In one possible implementation, the device further includes:

[0130] The storage module is used to transfer the target quantization parameters corresponding to each of the multiple input data blocks and the quantization result data corresponding to the quantization input data to the cache for storage.

[0131] In one possible implementation, the first-precision floating-point data type includes 32-bit or 16-bit floating-point numbers; the second-precision floating-point data type includes 8-bit floating-point numbers.

[0132] The modules of the quantization unit can be implemented by dedicated hardware circuits or by general-purpose processing hardware (such as field-programmable logic devices FPGA, ARM processors, etc.) combined with executable logic instructions. This application does not limit the implementation of these modules.

[0133] In one possible implementation, embodiments of this specification also provide an inverse quantization unit in a neural network processor, the inverse quantization unit comprising:

[0134] The data acquisition module is used to, during the inference process of the large language model, respond to the processor's dequantization instruction for the target network layer in the large language model, acquire first input data and second input data to be dequantized from the cache, and acquire first quantization parameters and first quantization block information corresponding to the first input data, and second quantization parameters and second quantization block information corresponding to the second input data from the cache; the first input data and the second input data are quantization result data after data quantization processing during the inference process, and both the first input data and the second input data are second-precision floating-point data types;

[0135] The second data segmentation module is used to segment the first input data based on the first quantization segmentation information to obtain multiple first data blocks; and to segment the second input data based on the second quantization segmentation information to obtain multiple second data blocks;

[0136] The dequantization processing module is used to perform matrix multiplication and dequantization processing on the plurality of first data blocks and the plurality of second data blocks according to the first quantization block information and the second quantization block information to obtain dequantization result data; the dequantization result data is a floating-point data type with a first precision; the first precision is higher than the second precision.

[0137] Furthermore, by setting up a core (i.e., dequantization unit) in the neural network processor to perform dequantization processing during the inference process of large language models, and this dequantization unit has data block function, matrix multiplication function, and high-precision accumulation function (e.g., multiplying by the corresponding quantization parameter), the dequantization processing does not need to frequently interact with the cache, nor does it need to frequently move data between multiple cores, which greatly reduces memory access and improves the dequantization processing efficiency of large models.

[0138] In one possible implementation, the inverse quantization processing module includes:

[0139] The quantization parameter determination submodule is used to determine, based on the first quantization parameter, a first target quantization parameter corresponding to each of the plurality of first data blocks; and to determine, based on the second quantization parameter, a second target quantization parameter corresponding to each of the plurality of second data blocks.

[0140] The data block combination submodule is used to combine any first data block with any second data block to determine multiple data block combinations, each data block combination including one first data block and one second data block;

[0141] The data block dequantization result acquisition submodule is used to perform matrix multiplication and dequantization processing on the first target data block and the second target data block in the target data block combination based on the first target quantization parameter corresponding to the first target data block and the second target quantization parameter corresponding to the second target data block in the target data block combination, to obtain the data block dequantization result; the target data block combination can be any of the data block combinations; the first target data block and the second target data block refer to the first data block and the second data block in the target data block combination.

[0142] The dequantization result data acquisition submodule is used to obtain the dequantization result data based on the dequantization results of multiple data blocks.

[0143] In one possible implementation, the data block dequantization result acquisition submodule is specifically used to perform matrix multiplication on the first target data block and the second target data block to obtain an intermediate matrix multiplication result; the intermediate matrix multiplication result is a floating-point data type of the second precision; and the intermediate matrix multiplication result is dequantized based on the first target quantization parameter and the second target quantization parameter to obtain the data block dequantization result.

[0144] In one possible implementation, the data block dequantization result acquisition submodule is further configured to multiply the intermediate matrix multiplication result by the first target quantization parameter and the second target quantization parameter to obtain the data block dequantization result.

[0145] The modules of the inverse quantization unit can be implemented by dedicated hardware circuits or by general-purpose processing hardware (such as field-programmable logic devices FPGA, ARM processors, etc.) combined with executable logic instructions. This application does not limit the implementation of these modules.

[0146] This specification also provides a neural network processor, which includes a quantization unit as described above and an inverse quantization unit as described above. This allows quantization processing to be performed in one core, and inverse quantization processing of matrix multiplication to be performed in another core. Furthermore, the quantization unit has a data block function. Compared to existing quantization cores that are limited by data size (e.g., block-wise) when receiving data to be quantized, the quantization unit in this application can receive the quantization input data to be quantized all at once, and then perform fine-grained quantization processing by dividing the quantization input data into blocks. This eliminates the need for frequent interaction with the cache, thereby achieving efficient online quantization processing for large models. The inverse quantization unit has data block functionality, matrix multiplication functionality, and high-precision accumulation functionality (e.g., multiplying by the corresponding quantization parameter), which eliminates the need for frequent interaction with the cache and frequent data transfer between multiple cores, greatly reducing memory access and improving the efficiency of inverse quantization processing for large models. Based on this, the neural network processor exemplified in this specification can improve the efficiency of the LLM inference process, thus making it applicable to a wider range of large model inference scenarios.

[0147] Figure 12 This is a block diagram illustrating an electronic device for data quantization or dequantization in a large language model, according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data quantization processing method or a data dequantization processing method in a large language model.

[0148] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a data quantization processing method in a large language model or a data dequantization processing method in a large language model as described in the embodiments of this disclosure.

[0150] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform a data quantization processing method or a data dequantization processing method in a large language model as described in this disclosure. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0151] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the data quantization processing method or the data dequantization processing method in a large language model as described in the embodiments of this disclosure.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0153] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0154] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data quantization processing method in a large-scale language model, characterized in that, The method, applied to a quantization unit in a neural network processor, includes: During the inference process of the large language model, in response to the processor's quantization instruction for the target network layer in the large language model, the quantized input data to be quantized is obtained from the cache; the quantized input data is a first-precision floating-point data type; the target network layer is any network layer in the large language model that needs to undergo data quantization processing. The quantized input data is divided into blocks according to the preset quantization block to obtain multiple input data blocks; Based on the target input data in each input data block, quantization parameter processing is performed to obtain the target quantization parameters corresponding to each of the multiple input data blocks; the target input data satisfies preset data statistical conditions. The input data in each input data block is quantized according to the target quantization parameters corresponding to each input data block to obtain the quantization result data corresponding to the quantized input data. The quantization result data includes the quantization result of each input data. The quantization result data is a floating-point data type with second precision, which is lower than the first precision.

2. The method according to claim 1, characterized in that, The preset data statistical condition is the maximum absolute value; the quantization parameter processing based on the target input data in each input data block to obtain the target quantization parameters corresponding to each of the multiple input data blocks includes: Determine the input data with the largest absolute value in each input data block, and use it as the target input data in each input data block; Obtain the data range corresponding to the second precision floating-point data type; Based on the target input data in each input data block and the data range, the target quantization parameters corresponding to each of the plurality of input data blocks are determined.

3. The method according to claim 1, characterized in that, Based on the target quantization parameters corresponding to each input data block, the input data in each input data block is quantized to obtain the quantization result data corresponding to the quantized input data, including: Divide the input data in each input data block by the target quantization parameter corresponding to each input data block to obtain the quantization result corresponding to each input data. Based on the quantization results corresponding to each input data, the quantization result data corresponding to the quantized input data is obtained.

4. The method according to claim 1 or 3, characterized in that, The quantization unit includes a data cache space, and the method further includes: Save the plurality of input data blocks to the data cache space; The step of performing data quantization processing on the input data in each input data block according to the target quantization parameters corresponding to each input data block to obtain the quantization result data corresponding to the quantized input data includes: In response to obtaining the target quantization parameters corresponding to the target input data block, input data in the target input data block is extracted from the data cache space; the target data input block is any one of the plurality of input data blocks. The input data in the target input data block is quantized according to the target quantization parameters corresponding to the target input data block to obtain the quantization result data corresponding to the quantized input data.

5. The method according to claim 1, characterized in that, The method further includes: The target quantization parameters corresponding to each of the multiple input data blocks and the quantization result data corresponding to the quantization input data are transmitted to the cache for storage.

6. The method according to claim 1, characterized in that, First-precision floating-point data types include 32-bit or 16-bit floating-point numbers; second-precision floating-point data types include 8-bit floating-point numbers.

7. A method for data dequantization processing in a large language model, characterized in that, The method, applied to an inverse quantization unit in a neural network processor, includes: During the inference process of the large language model, in response to the processor's dequantization instruction for the target network layer in the large language model, the processor retrieves the first input data and the second input data to be dequantized from the cache, and retrieves the first quantization parameter and the first quantization block information corresponding to the first input data, as well as the second quantization parameter and the second quantization block information corresponding to the second input data from the cache; the first input data and the second input data are the quantization result data after data quantization processing during the inference process, and both the first input data and the second input data are second-precision floating-point data types; The first input data is divided into blocks based on the first quantization block information to obtain multiple first data blocks; and the second input data is divided into blocks based on the second quantization block information to obtain multiple second data blocks. Based on the first quantization parameter and the second quantization parameter, matrix multiplication and dequantization are performed on the plurality of first data blocks and the plurality of second data blocks to obtain dequantization result data; the dequantization result data is a floating-point data type with a first precision; the first precision is higher than the second precision.

8. The method according to claim 7, characterized in that, Based on the first quantization parameter and the second quantization parameter, matrix multiplication and dequantization are performed on the plurality of first data blocks and the plurality of second data blocks to obtain dequantized result data, including: Based on the first quantization parameter, a first target quantization parameter corresponding to each of the plurality of first data blocks is determined; and based on the second quantization parameter, a second target quantization parameter corresponding to each of the plurality of second data blocks is determined. Combine any of the first data blocks with any of the second data blocks to determine multiple data block combinations, each data block combination including one of the first data blocks and one of the second data blocks; Based on the first target quantization parameter corresponding to the first target data block and the second target quantization parameter corresponding to the second target data block in the target data block combination, matrix multiplication and dequantization are performed on the first target data block and the second target data block to obtain the data block dequantization result; the target data block combination is any of the data block combinations; the first target data block and the second target data block refer to the first data block and the second data block in the target data block combination. The inverse quantization result data is obtained based on the inverse quantization results of multiple data blocks.

9. The method according to claim 8, characterized in that, The first target data block and the second target data block are subjected to matrix multiplication and inverse quantization based on the first target quantization parameter corresponding to the first target data block and the second target data block corresponding to the second target data block in the target data block combination, to obtain the data block inverse quantization result, including: Matrix multiplication is performed on the first target data block and the second target data block to obtain an intermediate matrix multiplication result; the intermediate matrix multiplication result is a floating-point data type of the second precision. Based on the first target quantization parameter and the second target quantization parameter, the intermediate result of the matrix multiplication is dequantized to obtain the dequantized result of the data block.

10. The method according to claim 9, characterized in that, The step of performing dequantization processing on the intermediate result of the matrix multiplication based on the first target quantization parameter and the second target quantization parameter to obtain the dequantized result of the data block includes: The intermediate result of the matrix multiplication is multiplied by the first target quantization parameter and the second target quantization parameter to obtain the inverse quantization result of the data block.

11. A quantization unit in a neural network processor, characterized in that, The quantization unit includes: The quantization input data acquisition module is used to, during the inference process of the large language model, in response to the processor's quantization instruction for the target network layer in the large language model, acquire the quantization input data to be quantized from the cache; the quantization input data is a first-precision floating-point data type; the target network layer is any network layer in the large language model that needs to undergo data quantization processing. The first data segmentation module is used to segment the quantized input data into multiple input data blocks according to the preset quantization block. The quantization parameter acquisition module is used to perform quantization parameter processing based on the target input data in each input data block to obtain the target quantization parameters corresponding to each of the multiple input data blocks; the target input data satisfies preset data statistical conditions. The quantization processing module is used to perform data quantization processing on the input data in each input data block according to the target quantization parameters corresponding to each input data block, so as to obtain the quantization result data corresponding to the quantized input data. The quantization result data includes the quantization result of each input data. The quantization result data is a floating-point data type with second precision, which is lower than the first precision.

12. An inverse quantization unit in a neural network processor, characterized in that, The inverse quantization unit includes: The data acquisition module is used to, during the inference process of the large language model, respond to the processor's dequantization instruction for the target network layer in the large language model, acquire first input data and second input data to be dequantized from the cache, and acquire first quantization parameters and first quantization block information corresponding to the first input data, and second quantization parameters and second quantization block information corresponding to the second input data from the cache; the first input data and the second input data are quantization result data after data quantization processing during the inference process, and both the first input data and the second input data are second-precision floating-point data types; The second data segmentation module is used to segment the first input data based on the first quantization segmentation information to obtain multiple first data blocks; and to segment the second input data based on the second quantization segmentation information to obtain multiple second data blocks; The dequantization processing module is used to perform matrix multiplication and dequantization processing on the plurality of first data blocks and the plurality of second data blocks according to the first quantization parameter and the second quantization parameter to obtain dequantization result data; the dequantization result data is a floating-point data type with a first precision; the first precision is higher than the second precision.

13. A neural network processor, characterized in that, The neural network processor includes the quantization unit as described in claim 11 and the dequantization unit as described in claim 12.

14. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 6 or the method as described in any one of claims 7 to 10.

15. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6 or the method as claimed in any one of claims 7 to 10.

16. A computer program product, characterized in that, Includes computer instructions, which, when executed by a processor, cause the computer to perform the method as described in any one of claims 1 to 6 or the method as described in any one of claims 7 to 10.

Images