Accelerated quantization and dequantization data processing methods, systems, products, and terminals for neural network inference

Abstract

This application provides a method, system, product, and terminal for accelerating quantization and dequantization data processing for neural network inference. It involves grouping the weight data of the current layer and adaptively determining the corresponding dequantization processing method based on a preset configuration. Zero-point adjustment, multiplication-accumulation, and scaling are performed on the weight data of the current layer. Simultaneously, through accumulation and requantization, the target result data is obtained and used as the activation data for the next layer. This invention achieves efficient utilization of hardware resources through refined control of the quantization and dequantization process, reuse of DOT operation units, and the application of dequantization bypass modes. It solves the technical problems of low dequantization efficiency and high power consumption caused by insufficient utilization of vector operation units and unreasonable execution timing in existing dequantization operations, while balancing the accuracy and computational efficiency of neural network inference, significantly improving hardware utilization and computational efficiency.

Description

Technical Field

[0001] This invention relates to the field of neural network inference data processing, and in particular to an accelerated quantization and dequantization data processing method, system, product, and terminal for neural network inference. Background Technology

[0002] In the field of deep learning, model compression is one of the key technologies for accelerating inference. By quantizing high-precision floating-point data into low-precision integer data (such as INT8, INT4, etc.), model storage overhead and data transmission latency can be effectively reduced, thereby improving the inference efficiency of deep learning models on terminal devices and edge devices. Dequantization, as the inverse operation of the quantization process, has the core function of restoring the quantized low-precision integer data into high-precision floating-point data (such as FP32, FP16, etc.), providing data support that meets the accuracy requirements for subsequent neural network inference operations. It is an indispensable key step in the inference process after deep learning model compression.

[0003] The implementation of the dequantization process can be expanded from three core dimensions: mathematical principles, hardware implementation, and framework optimization. Among them, the mathematical principles are the foundation of the dequantization operation, and its core formula is: r = s(q−z); where r represents the dequantized floating-point value, q represents the quantized low-precision integer value (e.g., INT8 type data), s represents the scaling factor (scale), the value of which is determined by the numerical range of the original floating-point data, and z represents the zero point, which is mainly used in asymmetric quantization scenarios to achieve zero point alignment between integer data and floating-point data, ensuring accurate precision during the dequantization process.

[0004] Regarding the setting of quantization parameters, existing technologies typically employ per-tensor or per-channel configuration. While tensor-based quantization is simple to implement, it struggles to adapt to local variations in the weight data's distribution; channel-based quantization offers higher accuracy but incurs significant parameter storage overhead. For group quantization, a finer-grained quantization method, existing inverse quantization implementations lack targeted hardware optimization solutions, resulting in low efficiency. Furthermore, existing technologies still exhibit significant efficiency bottlenecks in the implementation of inverse quantization functionality, specifically manifested in the following two core issues:

[0005] On the one hand, existing dequantization operations are mostly implemented using vector operation units. However, due to the inherent limitations of the vector length of the vector operation unit, its utilization rate is low, resulting in low overall efficiency of dequantization operations and making it unsuitable for the fast dequantization processing needs of large-scale low-precision data.

[0006] On the other hand, the execution timing of dequantization operations in existing technologies is unreasonable. Typically, matrix-related operations need to be completed through matrix operation units before dequantization is performed. This approach increases the overall latency of data processing, resulting in low coordination efficiency between dequantization operations and subsequent inference operations, which further restricts the inference acceleration effect of deep learning models. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, system, product and terminal for accelerating quantization and dequantization data processing for neural network inference, which solves the technical problem of low dequantization efficiency caused by insufficient utilization of vector operation units and unreasonable execution timing in existing dequantization operations.

[0008] To achieve the above and other related objectives, this application provides an accelerated quantization and dequantization data processing method for neural network inference. The method is characterized by being applied to an AI processing chip, which has a storage unit. The method includes: acquiring the weight data of the current layer of the neural network and its corresponding quantization parameters from the storage unit, and acquiring the activation data of the current layer; dividing the weight data of the current layer into multiple weight data groups according to a preset grouping size; determining the dequantization processing method of the weight data of the current layer according to a preset configuration; when the dequantization processing method of the weight data of the current layer is determined to be the first dequantization processing method, performing zero-point adjustment processing, multiplication and accumulation, and scaling processing on each weight data group sequentially based on the activation data and quantization parameters of the current layer to obtain the operation result corresponding to each weight data group; performing accumulation processing and requantization processing on all operation results to obtain target result data, which is used as the activation data of the next layer if a next layer exists; when the dequantization processing method of the weight data of the current layer is determined to be the second dequantization processing method, performing zero-point adjustment processing and scaling processing on each weight data group respectively to obtain the weight update data of the current layer weight data after dequantization.

[0009] In some embodiments of the first aspect of this application, the quantization parameters include zero-point parameters and scaling factors.

[0010] In some embodiments of the first aspect of this application, when the dequantization processing method for the weight data in the current layer is determined to be a first dequantization processing method, the zero-point adjustment processing, multiplication and accumulation processing, and scaling processing are performed sequentially on each weight data group based on the activation data and quantization parameters of the current layer, including: for each weight data group, performing zero-point adjustment processing based on the zero-point parameters corresponding to the weight data group to obtain adjusted weight data; the bit width of the adjusted weight data is not less than the bit width of the weight data; performing multiplication and accumulation operations on the adjusted weight data and the activation data to generate a multiplication and accumulation result corresponding to each weight data group; and scaling the multiplication and accumulation result based on the scaling factor corresponding to each weight data group to obtain the operation result corresponding to each weight data group.

[0011] In some embodiments of the first aspect of this application, the zero-point adjustment process for each weight data group based on the zero-point parameter corresponding to the weight data group includes: for the weight data in each weight data group, subtracting the zero-point parameter corresponding to the weight data group or adding the negative value of the zero-point parameter corresponding to the weight data group from the weight data based on the zero-point parameter corresponding to the weight data group.

[0012] In some embodiments of the first aspect of this application, scaling the multiplicative summation result based on the scaling factor corresponding to each weight data group includes: performing a multiplication operation on the multiplicative summation result corresponding to each weight data group and the scaling factor corresponding to that weight data group to obtain the corresponding operation result.

[0013] In some embodiments of the first aspect of this application, the accumulation and requantization of all computation results includes: performing at least two levels of accumulation processing on the computation results corresponding to all weight data groups according to a preset parallelism configuration; obtaining accumulated result data; wherein the first level of accumulation processing corresponds to the accumulation processing of the first parallelism multiply-accumulate-accumulate unit; the second level of accumulation processing corresponds to the accumulation processing of the second parallelism multiply-accumulate-accumulate unit; and requantizing the accumulated result data to obtain target result data.

[0014] In some embodiments of the first aspect of this application, the parallelism of the first parallelism multiply-accumulate unit is 128, and the parallelism of the second parallelism multiply-accumulate unit is 256.

[0015] To achieve the above and other related objectives, a second aspect of this application provides an accelerated quantization and dequantization data processing system for neural network inference, applied to an AI processing chip. The AI ​​processing chip has a storage unit, comprising: a data acquisition module for acquiring the weight data of the current layer of the neural network and its corresponding quantization parameters from the storage unit, and acquiring the activation data of the current layer; a data partitioning module for dividing the weight data of the current layer into multiple weight data groups according to a preset grouping size; a data judgment module for determining the dequantization processing method of the weight data of the current layer according to a preset configuration; and a first dequantization processing module for performing dequantization processing on the weight data of the current layer. When the first dequantization processing method is determined, based on the activation data and quantization parameters of the current layer, zero-point adjustment, multiplication and accumulation, and scaling are sequentially performed on each weight data group to obtain the operation result corresponding to each weight data group; all operation results are accumulated and requantized to obtain the target result data, which is used as the activation data of the next layer if there is a next layer; the second dequantization processing module is used to perform zero-point adjustment and scaling on each weight data group respectively when the dequantization processing method of the current layer weight data is determined to be the second dequantization processing method to obtain the weight update data of the current layer weight data after dequantization.

[0016] To achieve the above and other related objectives, a third aspect of the present invention provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the accelerated quantization and dequantization data processing method for neural network inference.

[0017] To achieve the above and other related objectives, a fourth aspect of the present invention provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the accelerated quantization and dequantization data processing method for neural network inference.

[0018] As described above, the accelerated quantization and dequantization data processing method, system, product, and terminal for neural network inference of this application have the following beneficial effects: This invention groups the weight data of the current layer and adaptively determines the corresponding dequantization processing method based on a preset configuration. It performs zero-point adjustment, multiplication-accumulation, and scaling processing on the weight data of the current layer. Simultaneously, through accumulation and requantization processing, it obtains the target result data and uses it as the activation data for the next layer. This invention achieves efficient utilization of hardware resources through refined control of the quantization and dequantization process, reuse of DOT operation units, and the application of dequantization bypass mode. It solves the technical problems of low dequantization efficiency and high power consumption caused by insufficient utilization of vector operation units and unreasonable execution timing in existing dequantization operations, and balances the accuracy and computational efficiency of neural network inference, significantly improving hardware utilization and computational efficiency. Attached Figure Description

[0019] Figure 1 The diagram shown is a flowchart illustrating an accelerated quantization and dequantization data processing method for neural network inference according to an embodiment of the present invention.

[0020] Figure 2 The diagram shows a flowchart of a process in which zero-point adjustment, multiplication and accumulation, and scaling are performed sequentially on each weighted data group according to an embodiment of the present invention.

[0021] Figure 3 The diagram shows a flowchart of an accelerated quantization and dequantization data processing method in which the weight data is in INT4 format, according to a specific embodiment of the present invention.

[0022] Figure 4 The diagram shows a flowchart of an accelerated quantization and dequantization data processing method in which the weight data is in INT8 format, according to a specific embodiment of the present invention.

[0023] Figure 5 The diagram shown is a structural schematic of an accelerated quantization and dequantization data processing system for neural network inference according to an embodiment of the present invention.

[0024] Figure 6 The diagram shown is an electronic terminal according to an embodiment of the present invention. Detailed Implementation

[0025] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0026] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the present invention. It should be understood that other embodiments may also be used. In the embodiments of the present invention, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function and effect, without limiting their order. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., are not necessarily different.

[0027] Furthermore, in the embodiments of the present invention, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0028] Furthermore, in this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0029] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0030] <1> Neural Processing Unit (NPU): An NPU is a processor specifically designed to accelerate neural network computations. Through hardware optimization, it efficiently executes core operations such as matrix multiplication and convolution, featuring high throughput and low power consumption. It is especially suitable for inference tasks in edge computing scenarios such as mobile phones and IoT devices.

[0031] <2> Tensor Processor (TPU): TPU is a dedicated processor launched by Google that is deeply optimized for tensor operations. Its core adopts a systolic array architecture, which can achieve extremely high computational efficiency in large-scale matrix calculations. It is mainly used to accelerate AI inference tasks based on frameworks such as TensorFlow.

[0032] <3> Graphics Processing Unit (GPU): A GPU is a graphics processing unit. Originally designed for parallel processing of graphics and image rendering tasks, it has thousands of small computing cores that enable it to perform matrix multiplication and convolution operations required for deep learning in a highly parallel manner. Therefore, it is also widely used as general-purpose accelerated computing hardware in neural network inference tasks.

[0033] The present invention provides a method, system, product, and terminal for accelerating quantization and dequantization data processing for neural network inference. This method involves grouping the weight data of the current layer and adaptively determining the corresponding dequantization processing method based on a preset configuration. Zero-point adjustment, multiplication-accumulation, and scaling processing are performed on the weight data of the current layer. Simultaneously, through accumulation and requantization processing, the target result data is obtained and used as the activation data for the next layer. The technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the invention.

[0034] like Figure 1 The diagram illustrates a flowchart of an accelerated quantization and dequantization data processing method for neural network inference according to an embodiment of the present invention. The accelerated quantization and dequantization data processing method for neural network inference in this embodiment includes the following steps:

[0035] Step S11: Obtain the weight data of the current layer of the neural network and its corresponding quantization parameters from the storage unit, and obtain the activation data of the current layer.

[0036] It should be noted that this application applies to AI processing chips, which refer to integrated circuits comprising dedicated neural network computing units, storage units, and quantization processing units. Examples include NPUs, TPUs, or GPUs that support AI acceleration. The neural network computing units are used to perform operations including multiplication-accumulation, matrix operations, or vector operations required for neural network model inference; the quantization processing units are used to perform operations including zero-point adjustment, scaling, dequantization, and requantization; and the storage units include on-chip storage units (such as a unified buffer (UB)) and / or external memory (such as DRAM or HBM).

[0037] The neural network model comprises multiple layers, which are processed layer by layer during inference. The current layer refers to the layer being processed within the neural network model. The activation data of the current layer is the input data for each layer's processing, and the weight data of the current layer is the corresponding weight data for each layer's processing. Specifically, the weight data used for neural network model inference and the corresponding quantization parameters are pre-stored in storage units, preferably in external memory. The activation data is the computational result data of the previous layer of the neural network model, stored in storage units. Preferably, during the pre-filling stage, the activation data is stored in external memory. During the decoding stage, if on-chip storage resources are sufficient, the activation data can be cached in on-chip storage units to reduce memory access overhead. For the input layer, the activation data is the original input data of the neural network model.

[0038] Specifically, the weight data of the current layer, the quantization parameters corresponding to the weight data, and the activation data of the current layer are obtained from the storage unit. The weight data supports data formats such as INT4 / U4 / INT8 / U8 / FP16; the quantization parameters include zero-point parameters and scaling factors. The zero-point parameters are used to adjust the offset of the quantized data to align the zero points of integer data with floating-point data, and the scaling factor is used to control the precision range of quantization, mapping integer data back to floating-point values. The coordinated configuration of the two can effectively ensure the numerical accuracy during the inference process of the neural network model.

[0039] In one embodiment of this application, the quantization parameters and weight data are stored correspondingly, with each weight data group corresponding to a set of zero-point parameters and scaling factors. To adapt to efficient hardware reading, the quantization parameters and corresponding weight data groups are arranged consecutively in the storage unit, facilitating the reading of the complete data group and its quantization parameters at once.

[0040] Step S12: Divide the weight data of the current layer into multiple weight data groups according to the preset group size.

[0041] In neural network model inference, the weight data is enormous (e.g., the LLaMA-7B model has 7 billion parameters). By grouping the weight data, we can achieve the following:

[0042] Parallel processing: Large-scale weighted data is split into multiple independent weighted data groups, which can be distributed to multiple parallel processing units for simultaneous calculation, greatly improving processing throughput;

[0043] Fine-grained quantization: Each weighted data group can be independently configured with quantization parameters (including zero-point parameters and scaling factors), which can better adapt to the local distribution characteristics of the weighted data and reduce the impact of outliers on the overall quantization and dequantization accuracy.

[0044] Hardware adaptation: The group size can be matched with the parallelism of the hardware processing unit (the multiply-accumulate operation unit in the corresponding dedicated neural network computing unit).

[0045] In this embodiment, the preset group size is a configurable parameter, supporting multiple group sizes to adapt to different neural network model characteristics and hardware resources. Typical group sizes include 32, 64, 128, 256, etc., preferably multiples of 64, to match the data path width of the hardware processing unit.

[0046] In one embodiment of this application, the group size is matched to the parallelism of the multiply-accumulate operation unit. For example:

[0047] When the group size is 64, it corresponds to a multiply-accumulate unit with a parallelism of 64 (such as DOT64).

[0048] When the group size is 128, it corresponds to a multiply-accumulate unit with a parallelism of 128 (such as DOT128).

[0049] When the group size is 256, it corresponds to a multiply-accumulate unit with a parallelism of 256 (such as DOT256).

[0050] Through this mapping, each weighted data group can be processed independently by a corresponding multiply-accumulate unit, achieving efficient parallel computing.

[0051] The partitioning operation is performed by the grouping processing unit on the chip. This unit extracts a specified number of weight elements sequentially from the continuously stored weight data, forming independent weight data groups, according to a preset group size configuration. If the total weight data of the current layer is not divisible by the preset group size, the last group can be processed using zero-padding or default value padding to achieve the complete group size, and the padding data will not affect the accuracy of subsequent quantization, multiplication, accumulation, and other operations.

[0052] Step S13: Determine the dequantization processing method for the weight data of the current layer according to the preset configuration.

[0053] Specifically, the preset configuration can be generated by the neural network compiler during the model deployment phase and embedded in the model metadata; or it can be dynamically set by the runtime scheduler according to the context of the current inference task.

[0054] The inverse quantization processing method includes:

[0055] The first dequantization processing method is suitable for forward inference layers (such as the Q / K / V projection layers of MLP and Attention), and its goal is to efficiently generate the target result data of the current layer. This processing method performs multiplication-accumulation-addition operations in the integer domain, which is not a complete dequantization, significantly reducing computation and memory access overhead.

[0056] The second dequantization processing method is suitable for scenarios that require high-precision weight data, such as the initialization of key-value cache (KVCache), model calibration, mixed precision fallback, or debugging analysis. Its goal is to generate weight update data in floating-point format for subsequent high-precision calculations.

[0057] By employing a dequantization processing method selection mechanism, this application can ensure the efficiency of conventional forward inference processing while being compatible with special high-precision requirements, achieving a balance between performance and flexibility. For example, in the Decode stage of the Large Language Model (LLM), the first method is used for more than 99% of the layers to maximize throughput; the second method is only temporarily enabled for the K / V projection layer when the K / V Cache is generated for the first time to ensure cache accuracy.

[0058] Step S14: When the dequantization processing method of the weight data of the current layer is determined to be the first dequantization processing method, based on the activation data and quantization parameters of the current layer, zero-point adjustment processing, multiplication and accumulation, and scaling processing are performed on each weight data group in sequence to obtain the operation result corresponding to each weight data group; all operation results are accumulated and requantized to obtain the target result data, which is used as the activation data of the next layer when there is a next layer.

[0059] In one embodiment of this application, as Figure 2 As shown, when the dequantization processing method for the weight data of the current layer is determined to be the first dequantization processing method, based on the activation data and quantization parameters of the current layer, the zero-point adjustment processing, multiplication and accumulation, and scaling processing are performed sequentially on each weight data group, including:

[0060] Step S141: For each weighted data group, perform zero-point adjustment processing based on the zero-point parameters corresponding to that weighted data group to obtain adjusted weighted data; the bit width of the adjusted weighted data is not less than the bit width of the original weighted data; the data format of the zero-point parameters is the same as the data format of the weighted data. For example, if the original weighted data is INT4 bit width, the adjusted bit width is INT8 or INT16; if the original weighted data is INT8 bit width, the adjusted bit width is INT16 or INT32.

[0061] In one embodiment of this application, zero-point adjustment processing for each weighted data group, based on the zero-point parameter corresponding to that weighted data group, includes: for the weight data within each weighted data group, subtracting the zero-point parameter corresponding to that weighted data group from the weight data, or adding the negative value of the zero-point parameter corresponding to that weighted data group to the weight data. The results of both methods are consistent, and the appropriate method can be flexibly selected based on the hardware operation logic. The purpose of this zero-point adjustment processing is to eliminate the offset introduced during quantization, restore the relative numerical relationship of the weight data, and ensure the accuracy of subsequent multiplication and addition operations. Furthermore, the entire adjustment process is completed within the integer domain, without introducing floating-point operation overhead, thus improving processing efficiency.

[0062] Step S142: Perform a multiplication and summation operation on the adjusted weight data and the activation data to generate a multiplication and summation result corresponding to each weight data group;

[0063] It should be noted that the multiply-accumulate operation is the core computational operation for neural network model inference. It mainly involves multiplying and accumulating the adjusted weight data after zero-point adjustment with the corresponding activation data element by element to generate the multiply-accumulate result for each weight data group.

[0064] The multiply-accumulate operation is performed by a multiply-accumulate array within a dedicated neural network computing unit. This array comprises multiple parallel multiply-accumulate units, each corresponding to a weight data group, enabling parallel processing of the data groups. The multiply-accumulate array supports various parallelism configurations, and the parallelism configuration matches the group size. For example, when the group size is 64, a parallelism configuration of 64 is used, ensuring that each weight data group can be processed independently by a corresponding multiply-accumulate unit.

[0065] During multiplication and accumulation operations, the data precision is progressively increased to prevent overflow and ensure calculation accuracy:

[0066] Input data: The adjusted weight data is multiplied by the corresponding activation data to generate the multiplication result; the bit width of the adjusted weight data is usually more than that of the weight data before adjustment (for example, INT4 weight data requires 5 to 6 bits after zero-point adjustment).

[0067] Accumulation calculation: Multiple multiplication results are accumulated, and the bit width is further expanded to finally obtain the multiplicative summation result; the bit width of the multiplicative summation result is usually in INT32 format.

[0068] Accuracy Guarantee: This step-by-step precision expansion mechanism ensures that no overflow occurs during the accumulation process, maintaining the numerical accuracy of the calculation results.

[0069] Step S143: Based on the scaling factor corresponding to each weighted data group, scale the multiplication and accumulation result to obtain the calculation result corresponding to each weighted data group.

[0070] In one embodiment of this application, scaling the multiplicative summation result based on the scaling factor corresponding to each weighted data group includes: performing a multiplication operation between the multiplicative summation result corresponding to each weighted data group and the scaling factor corresponding to that weighted data group to obtain the corresponding operation result.

[0071] It should be noted that the scaling process is the core step of the dequantization process, which is executed by the quantization processing unit of the AI ​​processing chip. Its main purpose is to convert the result of multiplication and summation in the integer field into floating-point data, restore the original numerical range of the weight data before quantization, and ensure the accuracy of neural network inference. During the multiplication operation, the result of multiplication and summation (usually in INT32 format) and the scaling factor (floating-point format, such as FP16, FP32) are multiplied in floating-point format, and the final result is floating-point data.

[0072] In one embodiment of this application, the process of accumulating and requantizing all computation results to obtain target result data and using it as activation data for the next layer includes:

[0073] Cumulative processing:

[0074] Based on the preset parallelism configuration, at least two levels of accumulation processing are performed on the calculation results corresponding to all weight data groups to obtain the accumulated result data. The first-level accumulation processing corresponds to the accumulation processing of the first parallelism multiply-accumulate unit (e.g., DOT128), used to initially accumulate the calculation results of the same batch. The second-level accumulation processing corresponds to the accumulation processing of the second parallelism multiply-accumulate unit (e.g., DOT256), used to globally summarize the first-level accumulation results, achieving efficient aggregation and accumulation of the final calculation results. The parallelism of the first and second parallelism multiply-accumulate units can be flexibly adjusted according to the group size and hardware resources, adapting to the group sizes mentioned above (e.g., 64, 128, 256).

[0075] The accumulation process is the process of merging the calculation results corresponding to all weighted data groups into a single result. Since the weighted data is divided into multiple independent data groups and processed separately, the results of each group need to be accumulated to obtain the complete calculation result.

[0076] Requantization:

[0077] The accumulated result data is then requantized to obtain the target result data.

[0078] The requantization process is performed by the quantization processing unit of the AI ​​processing chip, which converts the floating-point accumulation result data into low-precision integer data (such as INT4, INT8) to compress the data volume, reduce subsequent data transmission and storage overhead, and adapt to the activation data input requirements of the next layer.

[0079] The requantization process can be performed using either channel-based quantization or tensor-based quantization: in channel-based quantization, the accumulated result data of each output channel is quantized using independent quantization parameters; in tensor-based quantization, the tensor formed by all accumulated result data is quantized using shared quantization parameters, which can be flexibly selected according to the inference accuracy requirements of the current layer.

[0080] The flow of the target result data obtained after requantization includes two forms:

[0081] Direct pass mode: The target result data is directly written to the input buffer of the next layer as the activation data of the next layer, and the next layer is immediately triggered to start processing;

[0082] Store-and-forward mode: The target result data is written to the storage unit and read when needed by the next layer. This mode is suitable for scenarios such as residual connections and multi-branch networks.

[0083] In a specific embodiment of this application, the parallelism of the first parallelism multiplication and accumulation unit is 128, and the parallelism of the second parallelism multiplication and accumulation unit is 256. Preferably, the first parallelism of the first parallelism multiplication and accumulation unit matches the preset group size in step S12 (e.g., when the group size is 128, the first parallelism is 128), and the second parallelism of the second parallelism multiplication and accumulation unit is an integer multiple of the first parallelism, ensuring efficient connection between the two levels of accumulation and avoiding data redundancy. It should be noted that the above parallelism sizes (128 and 256) are merely illustrative examples and are not intended to limit the scope of protection of this application. Those skilled in the art can flexibly configure other parallelism values, such as 64 or 512, according to actual factors such as group size, hardware resources, and performance requirements.

[0084] Step S15: When the dequantization processing method of the current layer's weight data is determined to be the second dequantization processing method, zero-point adjustment processing and scaling processing are performed on each weight data group respectively to obtain the weight update data of the current layer's weight data after dequantization.

[0085] It should be noted that the second dequantization processing method differs from the first inference path. This method does not participate in activation data operations or perform multiplication and accumulation; it only performs dequantization on the weight data separately, generating high-precision weight update data. The second dequantization processing method is suitable for special scenarios that require floating-point weight data, such as the initialization of key-value cache (KV Cache) in large language models, model calibration, mixed precision rollback, or debugging analysis.

[0086] In one embodiment of this application, the second inverse quantization processing method includes:

[0087] First, for the weight data within each weight data group, zero-point adjustment is performed based on the zero-point parameter corresponding to that group. That is, the zero-point parameter is subtracted from the weight data, or the negative value of the zero-point parameter is added to obtain the adjusted weight data.

[0088] Secondly, based on the scaling factor corresponding to each weight data group, the adjusted weight data is scaled, that is, multiplied with the scaling factor to obtain the inverse quantized weight data of that group.

[0089] By concatenating and integrating the dequantized weight data of all weight data groups, the updated weight data of the current layer after dequantization is obtained.

[0090] The zero-point adjustment and scaling processes are both executed by the quantization unit of the AI ​​processing chip. The entire process is completed efficiently in both the integer and floating-point domains, without consuming the inference resources of the neural network computing unit. It can be executed in parallel with the inference process, improving the overall processing efficiency of the system. The obtained weight update data is high-precision floating-point data, which can be stored in the on-chip storage unit or external memory for subsequent weight updates, model fine-tuning, accuracy calibration, or backup inference.

[0091] The accelerated quantization and dequantization data processing method provided in this application has the following advantages compared with traditional dequantization calculation:

[0092] First, the dequantization operation logic is optimized, significantly saving hardware resources and power consumption. In simple dequantization (dequant only) mode, traditional solutions require multiple M×K×N dimension multiplication operations to obtain K dequantized results for K channels, resulting in high multiplier resource consumption and high power consumption. This application optimizes the operation timing. During Tensorcore computation, after the dot product operation in the K direction is completed, the scaling factor (SCALE) multiplication or format conversion (CVT) operation is performed, followed by subsequent DOT operations. Dequantization processing can be completed without increasing the number of multipliers, effectively saving multiplier hardware resources and reducing power consumption during data computation, thus improving the energy efficiency ratio of dequantization processing and adapting to the low power consumption requirements of high-frequency computation of large LLM models.

[0093] Secondly, it supports a bypass mode in dequantization mode, reducing data transfer latency and power consumption, and adapting to large LLM model scenarios. This application's dequantization processing supports a bypass mode, which can directly convert high-precision data to low-precision data and write the converted data to the storage unit (UB / SRAM). Compared to traditional solutions that require data conversion and transfer via a vector operation unit, this mode directly bypasses the vector operation unit, reducing the frequency and path of data transfer, effectively reducing latency during data storage and transmission, and simultaneously reducing power consumption caused by data transfer. This provides strong support for scenarios sensitive to data processing latency and power consumption, such as large LLM model inference, further improving the efficiency and energy efficiency of the entire neural network inference process.

[0094] To better understand the implementation scheme of accelerated quantization and dequantization data processing methods used for neural network inference, the following section combines... Figure 3 and Figure 4 The specific implementation examples are described below:

[0095] Example 1 (corresponding) Figure 3 ):

[0096] This embodiment applies to a 3D stacked AI processing chip. The memory units (DRAM / HBM) are interconnected closely with the neural network computing unit and quantization processing unit via through-silicon vias (TSVs) to achieve high-bandwidth, low-latency data access. In this embodiment, the weight data is stored and transmitted using the INT4 format.

[0097] Data reading (corresponding to step S11): Read the INT4 format weight data of the current layer and the quantization parameters (zero-point parameter zp, scaling factor scale) corresponding to each weight data group from the external DRAM, and obtain the activation data of the current layer. Among them, the zero-point parameter zp has the same format as the weight data, which is INT4 format.

[0098] Weight grouping (corresponding to step S12): Divide the INT4 weight data of the current layer into multiple weight data groups according to a preset grouping size. In this embodiment, the grouping size is matched with the hardware computing unit, and the grouping size is set to 64 to adapt to the DOT64 computing unit.

[0099] Determining the dequantization processing method (corresponding to step S13): According to the preset configuration, the first dequantization processing method is determined for the current layer, which is used to directly complete the inference of the current layer and output the activation data of the next layer.

[0100] INT4 format weighted data quantization and dequantization inference calculation (corresponding to step S14):

[0101] 1. Zero point adjustment process (corresponding to step S141)

[0102] Perform zero-point adjustment on the INT4 weight data within each weight data group: data_new = data – zp or equivalent operation: data_new = data + (-zp), converting the INT4 weights into adjusted weight data in INT5 format to avoid computational overflow.

[0103] 2. Multiplication and addition operation (corresponding to step S142)

[0104] The adjusted weight data and activation data in INT5 format are transmitted to the DOT64 arithmetic unit to perform 64-way parallel multiply-accumulate operations and output the multiply-accumulate result in INT32 format.

[0105] 3. Scaling process (corresponding to step S143)

[0106] The scaling process is performed by multiplying the INT32 format multiplicative summation result output by the DOT64 arithmetic unit by a scaling factor, thereby converting the integer field result data into floating-point field data and completing the inverse quantization.

[0107] 4. Two-level accumulation and requantization

[0108] The inverse quantization results of multiple weighted data groups are accumulated in two stages (first stage is DOT128, second stage is DOT256) to obtain the accumulated result data. Then, the quantization processing unit requantizes it to convert it into a low-precision integer (such as INT4 / INT8) to obtain the target result data, which is used as the activation data for the next layer.

[0109] The target result data is directly fed into the input buffer of the next layer through a high-speed path within the 3D chip, forming a continuous inference pipeline. This embodiment achieves high throughput and low latency accelerated inference in high-frequency LLM computing scenarios such as GEMV and GEMM. The above parallelism values ​​are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0110] Example 2 (corresponding) Figure 4 ):

[0111] This embodiment is also applied to 3D stacked AI processing chips. The memory units (DRAM / HBM) are interconnected closely with the neural network computing units and quantization processing units via through-silicon vias (TSVs) to achieve high-bandwidth, low-latency data access. In this embodiment, the weight data is stored and transmitted in INT8 format.

[0112] Data reading (corresponding to step S11): Read the INT8 format weight data of the current layer and the quantization parameters (zero-point parameter zp, scaling factor scale) corresponding to each weight data group from the external DRAM, and obtain the activation data of the current layer. Among them, the zero-point parameter zp has the same format as the weight data, which is INT8 format.

[0113] Weight grouping (corresponding to step S12): The INT8 weight data of the current layer is divided into multiple weight data groups according to a preset grouping size. In this embodiment, the grouping size is matched with the hardware computing unit, and the grouping size is set to 32 to adapt to the DOT32 computing unit. It should be noted that the DOT32 computing unit and the DOT64 computing unit are reused, and the working mode is dynamically switched according to the weight format and grouping size. When the weight is in INT8 format and the grouping size is 32, the computing unit is switched to DOT32 to ensure parallel processing efficiency and reasonable utilization of hardware resources.

[0114] Determining the dequantization processing method (corresponding to step S13): According to the preset configuration, the first dequantization processing method is determined for the current layer, which is used to directly complete the inference of the current layer and output the activation data of the next layer.

[0115] INT8 format weighted data quantization and dequantization inference calculation (corresponding to step S14):

[0116] 2. Zero point adjustment process (corresponding to step S141)

[0117] Perform zero-point adjustment on the INT8 weight data within each weight data group: data_new = data – zp or equivalent operation: data_new = data + (-zp), converting the INT8 weights into adjusted weight data in INT9 format to avoid computational overflow.

[0118] 2. Multiplication and addition operation (corresponding to step S142)

[0119] The adjusted weight data and activation data in INT9 format are transmitted to the DOT32 arithmetic unit to perform 32-way parallel multiply-accumulate operations and output the multiply-accumulate result in INT32 format.

[0120] 3. Scaling process (corresponding to step S143)

[0121] The scaling process is performed by multiplying the INT32 format multiplicative summation result output by the DOT32 arithmetic unit by a scaling factor, thereby converting the integer field result data into floating-point field data and completing the inverse quantization.

[0122] 4. Two-level accumulation and requantization

[0123] The inverse quantization results of multiple weighted data groups are accumulated in two stages (first stage is DOT128, second stage is DOT256) to obtain the accumulated result data. Then, the quantization processing unit requantizes it to convert it into a low-precision integer (such as INT4 / INT8) to obtain the target result data, which is used as the activation data for the next layer.

[0124] The target result data is directly fed into the input buffer of the next layer through a high-speed path within the 3D chip, forming a continuous inference pipeline. This embodiment achieves high throughput and low latency accelerated inference in high-frequency LLM computing scenarios such as GEMV and GEMM. The above parallelism values ​​are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0125] like Figure 5 The diagram illustrates the structure of an accelerated quantization and dequantization data processing system 500 for neural network inference according to an embodiment of the present invention. The accelerated quantization and dequantization data processing system for neural network inference in this embodiment includes: a data acquisition module 501, a data partitioning module 502, a data judgment module 503, a first dequantization processing module 504, and a second dequantization processing module 505.

[0126] The data acquisition module 501 is used to acquire the weight data of the current layer of the neural network and its corresponding quantization parameters from the storage unit, and to acquire the activation data of the current layer.

[0127] The data partitioning module 502 is used to divide the weighted data of the current layer into multiple weighted data groups according to a preset grouping size.

[0128] The data judgment module 503 is used to determine the dequantization processing method of the weight data of the current layer according to the preset configuration;

[0129] The first type of dequantization processing module 504 is used to perform zero-point adjustment processing, multiplication and accumulation, and scaling processing on each weight data group based on the activation data and quantization parameters of the current layer when the dequantization processing method of the weight data of the current layer is determined to be the first type of dequantization processing method, to obtain the operation result corresponding to each weight data group; to perform accumulation processing and requantization processing on all operation results to obtain the target result data and use it as the activation data of the next layer when there is a next layer.

[0130] The second type of dequantization processing module 505 is used to perform zero-point adjustment processing and scaling processing on each weight data group when the dequantization processing method of the current layer weight data is determined to be the second type of dequantization processing method, so as to obtain the weight update data of the current layer weight data after dequantization.

[0131] It should be noted that the implementation principle and process of the accelerated quantization and dequantization data processing system for neural network inference provided in this embodiment of the invention are similar to those of the accelerated quantization and dequantization data processing method for neural network inference described above, and will not be repeated here. The specific processes by which each module performs the corresponding steps described above have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0132] It should also be understood that the module division in the embodiments of the present invention is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of the present invention can be integrated into a single processor, exist as separate physical entities, or two or more modules can be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0133] like Figure 6 The diagram shown is a schematic of an electronic terminal provided in an embodiment of this application. The electronic terminal includes at least one processor 601, a memory 602, at least one network interface 603, and a user interface 605. The various components in the device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general will label all buses as bus systems.

[0134] The user interface 605 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0135] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0136] In this embodiment of the invention, the memory 602 is used to store various types of data to support the operation of the electronic terminal 600. Examples of this data include: any executable program for operation on the electronic terminal 600, such as the operating system 6021 and application program 6022; the operating system 6021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 6022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The method for testing the lateral positioning accuracy of agricultural machinery provided in this embodiment of the invention can be included in the application program 6022.

[0137] The methods disclosed in the above embodiments of the present invention can be applied to processor 601, or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in the form of software. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 601 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0138] In an exemplary embodiment, the electronic terminal 600 may be used to execute the aforementioned method by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0139] According to the method provided in the embodiments of the present invention, the present invention also provides a computer program product, the computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform... Figure 1 The embodiments shown are accelerated quantization and dequantization data processing methods for neural network inference in any of the embodiments.

[0140] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0141] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0146] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0147] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0149] In summary, the accelerated quantization and dequantization data processing method, system, product, and terminal for neural network inference provided by this invention group the weight data of the current layer and adaptively determine the corresponding dequantization processing method based on a preset configuration. This involves performing zero-point adjustment, multiplication-accumulation, and scaling processing on the weight data of the current layer. Simultaneously, through accumulation and requantization, the target result data is obtained and used as the activation data for the next layer. This invention achieves efficient utilization of hardware resources through refined control of the quantization and dequantization process, reuse of DOT operation units, and the application of dequantization bypass mode. It solves the technical problems of low dequantization efficiency and high power consumption caused by insufficient utilization of vector operation units and unreasonable execution timing in existing dequantization operations, while balancing the accuracy and computational efficiency of neural network inference, significantly improving hardware utilization and computational efficiency.

[0150] Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0151] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An accelerated quantization dequantization data processing method for neural network inference, characterized in that, Applied to an AI processing chip, wherein the AI ​​processing chip has a storage unit, the method includes: Obtain the weight data and corresponding quantization parameters of the current layer of the neural network from the storage unit, and obtain the activation data of the current layer; Divide the weight data of the current layer into multiple weight data groups according to a preset grouping size; The dequantization processing method for the weight data of the current layer is determined according to the preset configuration. When the dequantization processing method of the weight data in the current layer is determined to be the first dequantization processing method, based on the activation data and quantization parameters of the current layer, zero-point adjustment processing, multiplication and accumulation, and scaling processing are performed on each weight data group in sequence to obtain the operation result corresponding to each weight data group; all operation results are accumulated and requantized to obtain the target result data, which is used as the activation data of the next layer when there is a next layer. When the dequantization processing method for the current layer's weight data is determined to be the second dequantization processing method, zero-point adjustment processing and scaling processing are performed on each weight data group to obtain the weight update data of the current layer's weight data after dequantization.

2. The method for accelerated quantization dequantization data processing for neural network inference according to claim 1, wherein, The quantization parameters include zero-point parameters and scaling factors.

3. The method for accelerated quantization dequantization data processing for neural network inference according to claim 2, wherein, When the dequantization processing method for the weight data of the current layer is determined to be the first dequantization processing method, based on the activation data and quantization parameters of the current layer, zero-point adjustment, multiplication and accumulation, and scaling processing are performed sequentially on each weight data group, including: For each weighted data group, zero-point adjustment processing is performed based on the zero-point parameter corresponding to the weighted data group to obtain the adjusted weighted data; the bit width of the adjusted weighted data is not less than the bit width of the weighted data. Perform a multiplication and summation operation on the adjusted weight data and the activated data to generate a multiplication and summation result corresponding to each weight data group; Based on the scaling factor corresponding to each weighted data group, the multiplication and accumulation results are scaled to obtain the calculation result corresponding to each weighted data group.

4. The method for accelerated quantization dequantization data processing for neural network inference according to claim 3, wherein, For each weighted data group, zero-point adjustment processing based on the zero-point parameters corresponding to that weighted data group includes: For the weight data within each weight data group, based on the zero-point parameter corresponding to that weight data group, subtract the zero-point parameter corresponding to that weight data group from the weight data or add the negative value of the zero-point parameter corresponding to that weight data group to the weight data.

5. The accelerated quantization and dequantization data processing method for neural network inference according to claim 4, characterized in that, Scaling the multiplication and accumulation results based on the scaling factor corresponding to each weighted data group includes: The sum of the multiplications corresponding to each weighted data group is multiplied by the scaling factor corresponding to that weighted data group to obtain the corresponding calculation result.

6. The accelerated quantization and dequantization data processing method for neural network inference according to claim 1, characterized in that, The process of accumulating and requantizing all calculation results includes: According to the preset parallelism configuration, the calculation results corresponding to all weight data groups are subjected to at least two levels of accumulation processing to obtain the accumulated result data; wherein, the first level of accumulation processing corresponds to the accumulation processing of the first parallelism multiplication accumulation unit; the second level of accumulation processing corresponds to the accumulation processing of the second parallelism multiplication accumulation unit. The accumulated result data is then requantized to obtain the target result data.

7. The accelerated quantization and dequantization data processing method for neural network inference according to claim 6, characterized in that, The parallelism of the first parallelism multiply-accumulate unit is 128, and the parallelism of the second parallelism multiply-accumulate unit is 256.

8. A data processing system for accelerating quantization and dequantization for neural network inference, characterized in that, The system is applied to an AI processing chip, which has a storage unit on it, and includes: The data acquisition module is used to acquire the weight data of the current layer of the neural network and its corresponding quantization parameters from the storage unit, and to acquire the activation data of the current layer; The data partitioning module is used to divide the weighted data of the current layer into multiple weighted data groups according to a preset grouping size. The data judgment module is used to determine the dequantization processing method of the weight data of the current layer according to the preset configuration; The first dequantization processing module is used to perform zero-point adjustment, multiplication and accumulation, and scaling processing on each weight data group based on the activation data and quantization parameters of the current layer when the dequantization processing method of the weight data of the current layer is determined to be the first dequantization processing method, in order to obtain the operation result corresponding to each weight data group; and to perform accumulation processing and requantization processing on all operation results to obtain the target result data, which is used as the activation data of the next layer when there is a next layer. The second dequantization processing module is used to perform zero-point adjustment and scaling processing on each weight data group when the dequantization processing method of the current layer weight data is determined to be the second dequantization processing method, so as to obtain the weight update data of the current layer weight data after dequantization.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, causes the computer to implement the accelerated quantization and dequantization data processing method for neural network inference as described in any one of claims 1 to 6.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the accelerated quantization and dequantization data processing method for neural network inference as described in any one of claims 1 to 6.

Images