A pure integer quantization method of a visual transformer model and a related device

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a pure integer quantization method of a visual Transformer model and a related device; the pure integer quantization method of the visual Transformer model comprises the following steps: based on a selected visual Transformer model, linear components and nonlinear components are respectively quantized and integrated to obtain a pure integer quantization model; when the nonlinear components are quantized, integer approximation algorithms are used to respectively reconstruct integer Softmax, GELU and LayerNorm nonlinear operators to obtain reconstructed integer Softmax, GELU and LayerNorm operators; a tensor virtual machine compiler framework is used to perform operator packaging and computation graph optimization on the pure integer quantization model to obtain an optimized pure integer computation graph. The technical scheme disclosed by the application realizes full integer calculation and efficient deployment of the ViT model on the FPGA end side by reconstructing integer operators of the three types of nonlinear operators, namely Softmax, GELU and LayerNorm, and combining computation graph scheduling optimization.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically relates to a pure integer quantization method and related apparatus for a visual Transformer (VisionTransformer, ViT) model. Background Technology

[0002] With the rapid development of deep learning and computer vision technologies, ViT models have demonstrated outstanding performance in tasks such as image classification, object detection, and semantic segmentation. However, ViT models typically have a large parameter scale and complex computational structure, heavily relying on high-performance graphics processing units (GPUs) for floating-point operations, making it difficult to achieve efficient inference directly on resource-constrained embedded hardware platforms such as field-programmable gate arrays (FPGAs). Given this, for applications requiring real-time response, low-power operation, or edge intelligence deployment, such as smart cameras, industrial inspection, autonomous driving, and medical image analysis, the computational and storage overhead of ViT models has become a significant bottleneck restricting their practical application.

[0003] To improve the efficiency of ViT models on edge devices, researchers have proposed various model compression and acceleration methods, mainly including quantization, pruning, and distillation. Among these, quantization, by converting floating-point weights and activation maps into low-bit integer forms, can significantly reduce computational complexity and storage bandwidth requirements while maintaining model accuracy, making it the most practical lightweight quantization method currently available. However, the ViT model structure contains a large number of nonlinear operators, such as Softmax (Softmax Function), GELU (Gaussian Error Linear Units), and LayerNorm (Layer Normalization), which are mathematically difficult to directly quantize into integers. Interpretably, existing quantization methods typically require retaining some floating-point operations during the inference phase, forming a "mixed integer-floating-point computation" mode. This not only disrupts the integrity of the model's computation graph but also leads to frequent type conversions and memory synchronization overhead, significantly reducing inference efficiency and making it difficult to achieve truly pure integer computation inference. Furthermore, existing quantitative research focuses primarily on operator-level precision preservation, lacking computational graph scheduling and compilation optimization strategies for the hardware execution layer. For example, on FPGA platforms, due to the lack of customized mapping and graph-level scheduling mechanisms for integer operator structures, models cannot fully utilize the advantages of FPGA's parallel pipeline and on-chip memory, resulting in low overall computational resource utilization, insufficient throughput, and high latency. Summary of the Invention

[0004] The purpose of this invention is to provide a pure integer quantization method and related apparatus for visual Transformer models, thereby solving the technical problems of existing ViT models, such as the difficulty in integer quantization of nonlinear components, fragmented computation graphs, and low hardware resource utilization. The technical solution disclosed in this invention achieves full integer computation and efficient deployment of ViT models on the FPGA side by reconstructing three types of nonlinear operators—Softmax, GELU, and LayerNorm—into integer operators, combined with computation graph scheduling optimization. This achieves the goal of low-power, low-latency deployment and efficient inference acceleration of ViT models on resource-constrained FPGA platforms.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a pure integer quantization method for a visual Transformer model, comprising the following steps:

[0007] Based on the selected visual Transformer model, the linear and nonlinear components are quantized separately and then integrated to obtain a pure integer quantization model. When quantizing the nonlinear components, an integer approximation algorithm is used to reconstruct the three types of nonlinear operators, Softmax, GELU, and LayerNorm, into integers to obtain the reconstructed Softmax integer operator, GELU integer operator, and LayerNorm integer operator.

[0008] Using the Tensor Virtual Machine compiler framework, the pure integer quantization model is encapsulated by operators and the computation graph is optimized to obtain an optimized pure integer computation graph.

[0009] Specifically, when using an integer approximation algorithm to reconstruct Softmax into integers, the Softmax containing exponential operations is transformed into integer arithmetic operations through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, as well as normalization and integer division operations, thus obtaining the Softmax integer operator. When using an integer approximation algorithm to reconstruct GELU into integers, GELU is approximated as the product of the Sigmoid function and the input, thus obtaining the GELU integer operator. When using an integer approximation algorithm to reconstruct LayerNorm into integers, the standard deviation is approximated using an integer iteration method to replace the square root operation, thus obtaining the LayerNorm integer operator.

[0010] A further improvement to the technical solution of this invention lies in that, through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, and normalization and integer division operations, the Softmax operator containing exponential operations is transformed into integer arithmetic operations, thus obtaining the Softmax integer operator.

[0011] When performing exponential stabilization, the expression for exponential stabilization is:

[0012] ;

[0013] In the formula, for The corresponding Softmax operator operation value, The first element in the input vector of the Softmax operator i One floating-point element; The length of the input vector. i This represents the index of the element whose Softmax output value is currently being calculated. This represents the traversal index variable in the summation operation; for Fixed-point representation; This is the quantization scaling factor; For integer values ​​participating in the Softmax calculation, yes The quantized integer value, It is the maximum value after quantization of the input vector;

[0014] When performing exponential bottoming and shift approximation, the bottoming formula is used. Convert the exponent base to 2; use integer shifting to achieve a multiplication approximation, the expression is:

[0015] ;

[0016] In the formula, The approximation is, This represents a right shift operation;

[0017] When performing integer factorization and linear approximation, the exponential term is decomposed into an integer part and a fractional part, represented as:

[0018] ;

[0019] In the formula, For the integer part, For the decimal part, This is a scaling factor used to adjust the range of the decimal part;

[0020] The decimal part is approximated by a linear function, expressed as:

[0021] ;

[0022] In the formula, The integer constant after rounding. ;

[0023] When performing normalization and integer division operations, the expression is:

[0024] ;

[0025] ;

[0026] In the formula, Output an integer; As molecules; The denominator is ; This is an integer division operation; This is the output bit width; This is the quantization scaling factor for the output integer; = The integer value after the approximate multiplication operation;

[0027] The output of the Softmax integer operator is .

[0028] A further improvement to the technical solution of this invention lies in approximating GELU as the product of a Sigmoid function and the input, in the process of obtaining the GELU integer operator,

[0029] The expression that approximates GELU as the product of the Sigmoid function and the input is:

[0030] ;

[0031] In the formula, It is the input of GELU; The input is Operator output at time; The approximation is: It is a sigmoid function operation.

[0032] A further improvement to the technical solution of this invention lies in the fact that, in the process of obtaining the LayerNorm integer operator, the standard deviation is approximated using an integer iteration method to replace the square root operation.

[0033] The expression for approximating the standard deviation using the integer iteration method is:

[0034] ;

[0035] In the formula, It is the first The standard deviation results of the next iteration; It is the first The standard deviation results of the next iteration; For LayerNorm input; for The variance; This represents a right shift operation.

[0036] A further improvement to the technical solution of this invention lies in employing a tensor virtual machine compiler framework to perform operator encapsulation and computation graph optimization on the pure integer quantization model, thereby obtaining the optimized pure integer computation graph.

[0037] The pure integer quantization model is imported into the front end of the Tensor Virtual Machine compiler framework to generate the Relay IR expression, and all tensors are constrained to be of type int8 fixed-point format, carrying fixed-point scaling parameters;

[0038] The Softmax, GELU, and LayerNorm integer operators are encapsulated into Relay-callable integer primitives through the operator registration mechanism in the Tensor Virtual Machine compiler framework, and an integer computation definition is provided for each integer operator. After encapsulation, the corresponding nodes in the original computation graph are replaced with integer primitive calling nodes. Graph-level optimization is performed on the Relay computation graph containing integer primitives. Through operator fusion, constant folding, and memory layout reorganization, the original computation graph is rewritten into a unified integer data flow computation graph, resulting in an optimized pure integer computation graph.

[0039] A further improvement of the technical solution of the present invention is that, by using the Tensor Virtual Machine compiler framework to perform operator encapsulation and computation graph optimization on the pure integer quantization model, and obtaining the optimized pure integer computation graph, the process further includes automatic scheduling optimization after the computation graph optimization. In the automatic scheduling optimization process, the intermediate representations corresponding to each fusion operator in the unified integer data flow computation graph are used as input, and the AutoTVM intelligent scheduling search mechanism in the Tensor Virtual Machine compiler framework is used to automatically generate the optimal execution scheduling strategy for each operator.

[0040] AutoTVM abstracts the operator execution process into a parameterized execution template determined by a set of scheduling parameters, and uses the following cost function as the optimization objective:

[0041] ;

[0042] In the formula, θ represents the set of scheduling parameters; This represents the estimated computational delay of the operator under the current scheduling policy; This represents the estimated storage access latency caused by data memory access and cache access; This indicates the synchronization overhead introduced by pipeline synchronization or parallel control; , , These are the coefficient parameters used to balance the weighted relationship between computation cost, memory access cost, and synchronization cost.

[0043] A further improvement of the technical solution of the present invention is that, after obtaining the optimized pure integer computation graph, it further includes: compiling the optimized pure integer computation graph into FPGA executable logic.

[0044] A second aspect of the present invention provides a pure integer quantization system for a visual Transformer model, comprising:

[0045] The quantization integration unit is used to integrate linear and nonlinear components based on a selected visual Transformer model to obtain a pure integer quantization model. When quantizing nonlinear components, an integer approximation algorithm is used to reconstruct the three types of nonlinear operators, Softmax, GELU, and LayerNorm, into integers to obtain reconstructed Softmax integer operators, GELU integer operators, and LayerNorm integer operators.

[0046] The encapsulation and optimization unit is used to encapsulate operators and optimize the computation graph of the pure integer quantization model using the tensor virtual machine compiler framework, so as to obtain an optimized pure integer computation graph.

[0047] Specifically, when using an integer approximation algorithm to reconstruct Softmax into integers, the Softmax containing exponential operations is transformed into integer arithmetic operations through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, as well as normalization and integer division operations, thus obtaining the Softmax integer operator. When using an integer approximation algorithm to reconstruct GELU into integers, GELU is approximated as the product of the Sigmoid function and the input, thus obtaining the GELU integer operator. When using an integer approximation algorithm to reconstruct LayerNorm into integers, the standard deviation is approximated using an integer iteration method to replace the square root operation, thus obtaining the LayerNorm integer operator.

[0048] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a pure integer quantization method for a visual Transformer model as described in any one of the first aspects of the present invention.

[0049] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a pure integer quantization method for a visual Transformer model as described in any one of the first aspects of the present invention.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] To address the technical challenges of integerization, computational graph fragmentation, and low hardware resource utilization in deploying existing ViT models on FPGA platforms, this invention discloses a pure integer quantization method for visual Transformer models. This method can achieve integer approximation for nonlinear components in ViT models. Combined with operator encapsulation and computational graph scheduling optimization, it enables low-power, low-latency, and high-efficiency pure integer quantization inference and deployment of ViT models on FPGA platforms. Specifically, the technical solution of this invention reconstructs the quantization inference process of the visual Transformer model at the computational structure level. Specifically, it discloses a pure integer quantization inference implementation method for FPGA platforms. Unlike existing technical solutions that only perform low-bit representation of weights and activations while still retaining floating-point operators such as Softmax, GELU, and LayerNorm, this invention rewrites the mathematical calculation form of nonlinear operators into an equivalent or approximate integer form. It transforms the floating-point calculation process that depends on exponentiation, division, square root, and normalization into an integer calculation process consisting of hardware-friendly operations such as shifting, integer multiplication and addition, integer division, and table lookup or piecewise linear approximation. This allows each functional module of the visual Transformer to have a complete pure integer executable form at the operator level. Based on this, the technical solution of the present invention further starts from the overall structure of the computation graph, and performs unified encapsulation and fusion scheduling on various operators after integer reconstruction, constructing an end-to-end integer computation graph from input to output that does not contain any floating-point nodes. This completely eliminates the problems of integer-floating-point mixed computation, frequent type conversion, and computation graph fragmentation and pipeline interruption caused by the fallback of nonlinear operators to floating-point implementation in traditional quantization schemes. This enables the compiler to perform continuous operator fusion, cross-layer pipeline scheduling, and on-chip memory reuse optimization on the entire graph. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a pure integer quantization method for a visual Transformer model in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram illustrating the principle of a pure integer quantization method for a visual Transformer model in an embodiment of the present invention; illustratively, Figure 2 middle, I Q ,S Q These are the integer value of the Q (query) tensor after quantization and the quantization scaling factor, respectively. I K , S K These are the integer value of the K (bond) tensor after quantization and the quantization scaling factor, respectively. I A , S A These are the integer values ​​of the quantized attention features and the quantization scaling factor, respectively. This represents the intermediate integer value after the attention features have been processed by integer operators; INT32 / INT8 represent the integer bit width, where INT32 is the intermediate integer bit width during the calculation process, and INT8 is the final target bit width for quantization; I p Provide the integer input value for the Sigmoid function in GELU calculations;

[0055] Figure 3 This is a schematic diagram of a pure integer quantization system for a visual Transformer model in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0057] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0058] Please see Figure 1 The present invention provides a pure integer quantization method for a visual Transformer model, comprising the following steps:

[0059] Step 1: Based on the selected visual Transformer model, the linear and nonlinear components are quantized separately and then integrated to obtain a pure integer quantization model. When quantizing the nonlinear components, the integer approximation algorithm is used to reconstruct the three types of nonlinear operators, Softmax, GELU and LayerNorm, into integers to obtain the reconstructed Softmax integer operator, GELU integer operator and LayerNorm integer operator.

[0060] Step 2: Using the Tensor Virtual Machine (TVM) compiler framework, the pure integer quantization model obtained in Step 1 is encapsulated with operators and the computation graph is optimized to obtain the optimized pure integer computation graph.

[0061] Specifically, when using an integer approximation algorithm to reconstruct Softmax into integers, the Softmax function containing exponential operations is transformed into integer arithmetic operations through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, as well as normalization and integer division operations, thus obtaining the Softmax integer operator. When using an integer approximation algorithm to reconstruct GELU into integers, the GELU activation function is approximated as the product of the Sigmoid function and the input, thus obtaining the GELU integer operator. When using an integer approximation algorithm to reconstruct LayerNorm into integers, the standard deviation is approximated using an integer iteration method to replace the square root operation, thus obtaining the LayerNorm integer operator.

[0062] Please see Figure 2 In the specific exemplary technical solution of this invention, the specific process of transforming the Softmax function containing exponential operations into integer arithmetic operations through exponential stabilization, exponential base changing and shift approximation, integer factorization and linear approximation, and normalization and integer division operations is as follows:

[0063] When performing exponential stabilization, the expression is:

[0064] ;

[0065] In the formula, The first element in the input vector of the Softmax operator i A floating-point element, for The corresponding Softmax operator operation value; for Fixed-point representation; The length of the input vector. i This represents the index of the element whose Softmax output value is currently being calculated. This represents the traversal index variable in the summation operation; This is the quantization scaling factor; For integer values ​​participating in the Softmax calculation, yes The quantized integer value (i.e., the integer part of the fixed-point representation). It is the maximum value of the quantized input vector; interpretably, the maximum value is subtracted here to avoid overflow.

[0066] When performing exponential bottoming and shift approximation, the bottoming formula is used. Convert the base of the exponent to 2; because Multiplication can be approximated using integer shifting, as shown in the expression:

[0067] ;

[0068] In the formula, This represents a right shift operation.

[0069] When performing integer factorization and linear approximation, to accommodate shift operations, the exponent term is decomposed into an integer part and a fractional part, represented as:

[0070] ;

[0071] In the formula, For the integer part, The decimal part.

[0072] The decimal part is approximated by a linear function, expressed as:

[0073] ;

[0074] In the formula, .

[0075] The above steps have been completed. The approximation of the molecules in the calculation formula, i.e. ,in = The integer value after approximation and multiplication.

[0076] When performing normalization and integer division operations, the numerator and denominator are scaled by the same factor before integer division. The expression is:

[0077] ;

[0078] ;

[0079] In the formula, This is the output bit width, used to control the range and precision of the output integer values; This is the quantization scaling factor corresponding to the output integer value; For integer division operations, As molecules, The denominator is .

[0080] The final output is .

[0081] In the specific exemplary technical solution of this invention, the step of approximating the GELU activation function as the product of the Sigmoid function and the input to obtain the GELU integer operator is as follows:

[0082] The expression that approximates the GELU activation function as a Sigmoid form is:

[0083] ;

[0084] In the formula, It is the input of GELU. It is input Operator output at time, It is a sigmoid function operation.

[0085] In the further preferred technical solution, Calculation of exponents in numerator and denominator The Softmax integer operator exponent module can be directly reused. An approximate reconstruction method.

[0086] In the specific exemplary technical solution of this invention, the step of obtaining the LayerNorm integer operator by approximating the standard deviation using an integer iteration method to replace the square root operation is as follows:

[0087] Since LayerNorm involves square root operations, it cannot be directly integerized; in the technical solution of this invention embodiment, an integer approximation of the standard deviation calculation is achieved through an integer iteration method, expressed as:

[0088] ;

[0089] In the formula, It is the first The standard deviation results of the next iteration It is the first The standard deviation results of the next iteration For LayerNorm input, For calculation The variance. Interpretatively, the above iterative process is fixed at 10 iterations to ensure convergence.

[0090] In this embodiment of the invention, the pure integer quantized ViT model obtained in step 1 is encapsulated with integer operators and its computational graph structure is optimized using the TVM compiler framework to construct a unified end-to-end pure integer computational graph. This process follows the main thread of "model import, intermediate representation, operator rewriting, and graph-level optimization," with the output of each stage serving as the input for subsequent stages, forming a standardized intermediate computational representation that can be used for subsequent scheduling and deployment.

[0091] Specifically, the pure integer quantization model output from step 1 is first imported into the TVM frontend to generate the corresponding Relay IR representation. All tensors in the model are constrained to be of type int8 fixed-point format, carrying corresponding fixed-point scaling parameters. This ensures that no floating-point tensors or floating-point operators are introduced into the computation graph at the intermediate representation layer. Next, the integer operator encapsulation stage begins. The integer Softmax, integer GELU, and integer LayerNorm operators reconstructed in step 1 are encapsulated into Relay (high-level intermediate representation) callable integer primitives through the TVM operator registration mechanism. Each operator is provided with a corresponding integer computation definition, enabling it to participate in graph-level transformations as a computation graph node and in subsequent optimization processes as a basic operator unit. After encapsulation, the corresponding nodes in the original computation graph are replaced with the aforementioned integer primitive calling nodes, thus eliminating the dependency on floating-point operators at the operator level. Based on this, graph-level optimization is performed on the Relay computation graph containing the above integer primitives. Through transformations such as operator fusion, constant folding and memory layout reorganization, the original computation graph is rewritten into a unified integer data flow computation graph that is topologically equivalent but structurally more regular. With the elimination of floating-point nodes as a constraint, it is ensured that the optimized computation graph no longer contains any floating-point operators or integer-to-floating-point type conversion nodes.

[0092] After completing the construction of the pure integer computation graph, the automatic scheduling optimization stage is entered. This stage takes the intermediate representations corresponding to each fusion operator in the unified integer computation graph as input and uses the AutoTVM intelligent scheduling search mechanism in TVM to automatically generate the optimal or near-optimal execution scheduling strategy for each operator.

[0093] AutoTVM abstracts the operator execution process into a set of scheduling parameters. The determined parameterized execution template is optimized using the following cost function:

[0094] ;

[0095] In the formula, θ represents the set of scheduling parameters used to describe scheduling decisions such as cyclic block size, expansion factor, parallel partitioning method, and cache insertion position; This represents the estimated computational delay of the operator under the current scheduling policy; This represents the estimated storage access latency caused by data memory access and cache access; This indicates the synchronization overhead introduced by pipeline synchronization or parallel control; , , These are the coefficient parameters used to balance the weighted relationship between computation cost, memory access cost, and synchronization cost.

[0096] During the search process, AutoTVM evaluates and filters different combinations of scheduling parameters based on the cost model, and automatically selects the scheduling configuration that minimizes the cost function while meeting the resource constraints of the target platform. At the same time, the automatic scheduling process jointly considers the overall structure after the fusion of integer operators in the search space, and performs collaborative optimization on the computation order, data reuse method and intermediate result cache location within the fused operators. This achieves cache reuse optimization, computation and memory access pipeline overlap and parallel granularity adaptive adjustment at the scheduling level, so that the computation latency and memory access latency in the overall execution process are balanced.

[0097] In a further preferred embodiment of the present invention, the method further includes: a computation graph compilation and hardware mapping step; wherein the optimized Relay graph is compiled into FPGA executable logic (TIR / VTA instruction stream). The compiler generates corresponding shifter, multiply-accumulator, and integer divider modules based on the quantization bit width and scaling factor, thereby achieving automatic mapping and efficient execution of integer operators on the FPGA.

[0098] This invention proposes a pure integer quantization framework for visual Transformers. By reconstructing nonlinear functions using integer approximations, combined with operator encapsulation and computation graph scheduling optimization, it achieves integer inference computation throughout the entire Transformer process. Specifically, this invention designs integer approximation algorithms for Softmax, GELU, and LayerNorm, achieving accurate integer approximations through hardware-friendly arithmetic operations such as shifting and integer division. Based on the TVM compilation framework, it implements operator encapsulation, fusion, and scheduling optimization for the integer quantization model, enabling integer operators to form a continuous and efficient computation graph on the FPGA, avoiding graph segmentation caused by floating-point operator insertion. Through instruction pipeline refactoring, bit-width adaptation, and on-chip memory scheduling, it achieves hardware mapping of integer operators, fully utilizing the parallel computing and low-power characteristics of the FPGA to realize efficient inference of the visual Transformer model.

[0099] In this embodiment of the invention, since the integer operators are designed from the outset to have a pipeline-friendly structure adapted to FPGAs, such as fixed-point multiplication-addition, shifting, and table lookup, the invention can further combine bit-width planning, instruction pipeline reconfiguration, and on-chip memory scheduling during the deployment phase to map each sub-module of the visual Transformer into a continuous streaming data path structure, thereby fully leveraging the parallel computing and low-power characteristics of FPGAs. Through the above-mentioned collaborative reconfiguration from the mathematical form of operators, computation graph structure to hardware mapping method, the invention achieves pure integer, low-latency, high-throughput, and low-power inference of the visual Transformer on the FPGA platform without introducing any floating-point operations, effectively solving the problems of low efficiency and poor hardware adaptability caused by mixed-precision computation in the prior art.

[0100] To complement the aforementioned pure integer quantization method, this invention presents a modular FPGA acceleration device that employs a pipelined parallel structure and on-chip memory reuse mechanism to achieve low-power, high-throughput integer inference. The modular FPGA acceleration device mainly includes the following functional modules:

[0101] The data input module is responsible for loading quantization parameters and input features, and supports INT4 / INT8 data alignment and DMA high-speed transfer.

[0102] The integer operator execution module, serving as the core of the device, consists of multiple parallel sub-units. These sub-units include: an integerized Softmax unit for implementing integer exponent swapping and normalization in Softmax; an integerized GELU unit for implementing the activation function based on shift approximation and integer sigmoid; and an integerized LayerNorm unit for calculating the standard deviation using integer iteration to achieve integer normalization. Each sub-unit executes in a pipelined manner, with data transferred directly on-chip without external memory access.

[0103] The control and scheduling module is used to control the execution order of operators, task distribution and parallel scheduling based on the scheduling table generated by the compiler, and supports operator fusion and asynchronous execution.

[0104] The on-chip storage and quantization parameter cache module is used to implement alternating input / output processing using a double buffering mechanism, and to cache scaling factors and zero-point parameters to improve memory access efficiency.

[0105] The result output module is used to perform inverse quantization and data aggregation of integer results, and output them to the host or external storage.

[0106] In principle, the visual Transformer model contains nonlinear operators such as Softmax, GELU, and LayerNorm. These operators involve exponential, division, or normalization operations, which are difficult to express precisely in the integer domain. Existing quantization methods typically use floating-point substitution or lookup table approximation, resulting in the model still requiring some floating-point operations during the inference phase, thus compromising the consistency of the computation graph and the integrity of integerization. FPGA on-chip resources (such as LUTs, DSPs, and BRAMs) are limited, and existing operator mapping schemes often do not fully consider the logic multiplexing and pipeline design of integer operators, leading to low utilization of computing units and failing to realize the potential of FPGAs in parallel and low-power computing. Most existing deep learning compilation frameworks primarily optimize for GPUs and do not design dedicated scheduling strategies for the characteristics of integer operators and the FPGA hardware structure, resulting in low operator fusion, high memory access latency, and insufficient data reuse, limiting the overall acceleration effect. In summary, the core problem addressed by this invention is how to achieve efficient computation of quantized visual Transformer models on an FPGA. The technical solution of this invention involves integer reconstruction of the three nonlinear operators—Softmax, GELU, and LayerNorm—from an algorithmic perspective. This improves the computational efficiency of the operators themselves while avoiding mixed-precision segmentation of the computation graph. Furthermore, to ensure practical deployment on an FPGA, steps such as TVM tuning and hardware mapping are also included.

[0107] In a specific embodiment of the present invention, it is implemented on an acceleration platform based on a Xilinx ZCU104 FPGA. The host uses the Ubuntu 22.04 operating system, the compiler uses TVM version 0.9.dev0, the hardware logic synthesis uses Vivado2023.1, the integer quantization inference model is deployed and executed on the FPGA, the model uses the ViT-S / 16 baseline architecture, and the input image resolution is 224×224.

[0108] In a specific embodiment of the present invention, the implementation process of the visual Transformer pure integer quantization method is as follows:

[0109] Step (1), linear component quantization, includes: performing symmetric fixed-point quantization of the linear weights and activations of the ViT model, using 8-bit integers for representation, i.e. In the formula, It is an input floating-point parameter. This is the quantization scaling factor. It's a rounding operation. The integer value obtained by quantization, yes The absolute value of , max is the operation to take the maximum value, k=8 is the quantization bit width.

[0110] Step (2): For the three types of nonlinear operators Softmax, GELU and LayerNorm, the integer approximation algorithm proposed in this invention is used to reconstruct them into integers.

[0111] Step (3), computation graph compilation and scheduling optimization; wherein,

[0112] 1) Integer operator encapsulation and Relay IR optimization include: Relay IR based on the TVM framework encapsulates the reconstructed integer operators. First, integer operators are defined and registered, mapping the integer form of nonlinear components to Relay primitives, represented as:

[0113] ;

[0114] In the formula, Input as an integer. For the input scaling factor, This is the output bit width; It is a Relay primitive that approximates the function of softmax. It is a Relay primitive with a function similar to GELU; yes The output, yes The output.

[0115] Each operator, after registration, forms a callable module, supporting cross-layer scheduling and fusion. Then, the reconstructed integer operators are embedded into the Relay computation graph of the ViT model. Memory access frequency is reduced through operator fusion, constant folding, and memory layout optimization, as shown below:

[0116] ;

[0117] In the formula, This is the original Relay calculation diagram. This refers to operations such as operator fusion, constant folding, and memory layout optimization to reduce memory access frequency. This is the optimized Relay calculation graph.

[0118] Finally, by statically analyzing the topology of the computation graph, the data flow dependencies between integer operators are adjusted to ensure that the computation paths are continuous and parallel, thus avoiding graph cutting caused by mixed integer-floating-point computations.

[0119] 2) The computation scheduling and automatic optimization based on AutoTVM include: after the integer operators are encapsulated, the AutoTVM automatic scheduling module is used to perform multi-level optimization of the computation graph; among them, operator-level scheduling search is first performed, and the execution mode of each operator is defined by the scheduling template, such as block, vectorization, parallel and pipeline strategies.

[0120] AutoTVM minimizes the cost function of the following expression:

[0121] ;

[0122] In the formula, θ represents the set of scheduling parameters used to describe scheduling decisions such as cyclic block size, expansion factor, parallel partitioning method, and cache insertion position; This represents the estimated computational delay of the operator under the current scheduling policy; This represents the estimated storage access latency caused by data memory access and cache access; This indicates the synchronization overhead introduced by pipeline synchronization or parallel control; , , These are the coefficient parameters used to balance the weighted relationship between computation cost, memory access cost, and synchronization cost.

[0123] Simultaneously, cache remapping is performed using the FPGA's on-chip memory structure (BRAM, DSP), expressed as:

[0124] ;

[0125] In the formula, This represents the overall storage access reuse benefit metric under the current scheduling and cache mapping strategy; Indexes representing different data objects; Indicates the first The number of memory accesses or the weight of memory access costs for each data object; Indicates the first The reuse ratio coefficient of each data object in the on-chip cache; summation is used to accumulate statistics on all data objects involved in cache mapping.

[0126] Finally, the parallelism P and pipeline depth L of the FPGA execution unit are jointly optimized to maximize throughput:

[0127] ;

[0128] In the formula, , This indicates a maximize operation.

[0129] In this embodiment of the invention, the implementation process of the FPGA acceleration device includes:

[0130] (1) Device structure design: The FPGA acceleration device structure includes: a data input module, an integer operator execution module, a control and scheduling module, an on-chip storage management module, a quantization parameter cache module, and a result output module. Among them, the data input module receives input data from the host through a DMA (Direct Memory Access) controller and aligns it to the internal cache according to the bit width. The integer operator execution module consists of three integer sub-units. Each sub-unit independently performs integer calculations and is connected in series through an AXI (Advanced eXtensible Interface) bus to realize streaming data transmission. The control and scheduling module implements operator-level scheduling through an FSM (Finite State Machine) state machine and controls the execution order and parallel granularity according to the scheduling table S generated by the TVM compilation. The on-chip storage module adopts a double buffer mechanism for data alternating access to reduce memory access pauses. The output module performs dequantization on the result and writes the output back to DDR (Double Data Rate Synchronous Dynamic Random Access Memory).

[0131] (2) Execution process and operator timing: In the Vision Transformer, each layer module is executed sequentially, and the operators are in a serial relationship. All modules in the device are independent of each other in terms of hardware structure, but the calculation order strictly follows the hierarchical topology of ViT, forming an execution strategy of serial execution between layers and parallel execution within layers.

[0132] Specifically, the execution process is as follows:

[0133] Input→Linear / MatMul→Shiftmax(Attention)→MatMul / Residual→ShiftGELU(MLP)→I-LayerNorm→Next Layer. That is, the matrix multiplication, activation, normalization and other operations of each layer are triggered sequentially on the FPGA. After the previous layer is completed, the output is buffered to on-chip storage and used as the input of the next layer.

[0134] Intra-layer parallelism is achieved through the operator scheduling module, including: the same matrix multiplication unit can be executed in parallel on multiple tiles; the exponential shift operation in integer Softmax and integer GELU can be performed synchronously on multiple data channels; and the mean and variance calculation of integer LayerNorm is carried out in parallel through a multi-core accumulator.

[0135] (3) Computation scheduling and storage optimization: The control and scheduling module automatically schedules the startup timing, memory access pointer and data channel of each operator according to the optimization parameters generated by AutoTVM.

[0136] The FPGA employs a hierarchical caching structure: Level 1 cache (L1) stores operator input / output tensors; Level 2 cache (L2) stores weights and intermediate values; this double-buffering mechanism allows data loading and computation to occur alternately. During calculation, Load the next batch of data.

[0137] In the performance evaluation of this embodiment of the invention, accuracy evaluation and speed evaluation are performed. Regarding the accuracy evaluation, on the ImageNet Dataset dataset, the accuracy of the original model is 81.39%, and the accuracy of the model quantized by the technical solution of this embodiment of the invention is 81.27%, with only a 0.12% performance loss. Regarding the speed evaluation, on the ZCU104 FPGA, the frame rate of the original model is 38.9 FPS, and the model quantized by this embodiment of the invention reaches 79.3 FPS.

[0138] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0139] Please see Figure 3 In this embodiment of the invention, a pure integer quantization system for a visual Transformer model is provided, comprising:

[0140] The quantization integration unit is used to integrate linear and nonlinear components based on a selected visual Transformer model to obtain a pure integer quantization model. When quantizing nonlinear components, an integer approximation algorithm is used to reconstruct the three types of nonlinear operators, Softmax, GELU, and LayerNorm, into integers to obtain reconstructed Softmax integer operators, GELU integer operators, and LayerNorm integer operators.

[0141] The encapsulation and optimization unit is used to encapsulate operators and optimize the computation graph of the pure integer quantization model using the tensor virtual machine compiler framework, so as to obtain an optimized pure integer computation graph.

[0142] Specifically, when using an integer approximation algorithm to reconstruct Softmax into integers, the Softmax containing exponential operations is transformed into integer arithmetic operations through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, as well as normalization and integer division operations, thus obtaining the Softmax integer operator. When using an integer approximation algorithm to reconstruct GELU into integers, GELU is approximated as the product of the Sigmoid function and the input, thus obtaining the GELU integer operator. When using an integer approximation algorithm to reconstruct LayerNorm into integers, the standard deviation is approximated using an integer iteration method to replace the square root operation, thus obtaining the LayerNorm integer operator.

[0143] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to execute the pure integer quantization method of a visual Transformer model.

[0144] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the pure integer quantization method for the visual Transformer model in the above embodiments.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A pure integer quantization method for a visual Transformer model, characterized in that, Includes the following steps: Based on the selected visual Transformer model, the linear and nonlinear components are quantized separately and then integrated to obtain a pure integer quantized model. When quantizing the nonlinear components, integer approximation algorithms are used to reconstruct the three types of nonlinear operators, Softmax, GELU, and LayerNorm, into integers, resulting in reconstructed Softmax, GELU, and LayerNorm integer operators. The visual Transformer model is used for image classification, object detection, or semantic segmentation tasks. Using the Tensor Virtual Machine compiler framework, the pure integer quantization model is encapsulated by operators and the computation graph is optimized to obtain an optimized pure integer computation graph. Specifically, when using an integer approximation algorithm to reconstruct Softmax into integers, the Softmax integer operator is obtained by transforming the exponentially-containing Softmax into integer arithmetic operations through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, as well as normalization and integer division operations. When using an integer approximation algorithm to reconstruct GELU into integers, GELU is approximated as the product of the Sigmoid function and the input, thus obtaining the GELU integer operator. When using an integer approximation algorithm to reconstruct LayerNorm into integers, the standard deviation is approximated using an integer iteration method to replace the square root operation, thus obtaining the LayerNorm integer operator. in, Using the Tensor Virtual Machine compiler framework, the pure integer quantization model is encapsulated with operators and the computation graph is optimized to obtain the optimized pure integer computation graph. In this process, the pure integer quantization model is imported into the front end of the Tensor Virtual Machine compiler framework to generate a Relay IR expression, and all tensors are constrained to be of type int8 fixed-point format, carrying fixed-point scaling parameters. The Softmax integer operator, GELU integer operator, and LayerNorm integer operator are encapsulated into Relay-callable integer primitives through the operator registration mechanism in the Tensor Virtual Machine compiler framework, and an integer computation definition is provided for each integer operator. After encapsulation, the corresponding nodes in the original computation graph are replaced with integer primitive calling nodes. Graph-level optimization is performed on the Relay computation graph containing integer primitives. Through operator fusion, constant folding, and memory layout reorganization, the original computation graph is rewritten into a unified integer dataflow computation graph, resulting in the optimized pure integer computation graph. Using the Tensor Virtual Machine compiler framework, the pure integer quantization model is encapsulated into operators and the computation graph is optimized to obtain the optimized pure integer computation graph. Following computation graph optimization, automatic scheduling optimization is also included. In this automatic scheduling optimization process, the intermediate representations corresponding to each fused operator in the unified integer data flow computation graph are used as input. The AutoTVM intelligent scheduling search mechanism in the Tensor Virtual Machine compiler framework automatically generates the optimal execution scheduling strategy for each operator. AutoTVM abstracts the operator execution process into a parameterized execution template determined by a set of scheduling parameters, and uses the following cost function as the optimization objective: ； In the formula, θ represents the set of scheduling parameters; This represents the estimated computational delay of the operator under the current scheduling policy; This represents the estimated storage access latency caused by data memory access and cache access; This indicates the synchronization overhead introduced by pipeline synchronization or parallel control; , , These are coefficient parameters used to balance the weighted relationship between computation cost, memory access cost, and synchronization cost; After obtaining the optimized pure integer computation graph, the process also includes compiling the optimized pure integer computation graph into FPGA executable logic.

2. The pure integer quantization method for a visual Transformer model according to claim 1, characterized in that, Through exponential stabilization, exponential base shifting and approximation, integer factorization and linear approximation, as well as normalization and integer division operations, the Softmax integer operator is transformed into integer arithmetic operations. When performing exponential stabilization, the expression for exponential stabilization is: ； In the formula, for The corresponding Softmax operator operation value, The first element in the input vector of the Softmax operator i One floating-point element; The length of the input vector. i This represents the index of the element whose Softmax output value is currently being calculated. This represents the traversal index variable in the summation operation; for Fixed-point representation; This is the quantization scaling factor; For integer values ​​participating in the Softmax calculation, yes The quantized integer value, It is the maximum value of the quantized input vector; When performing exponential bottoming and shift approximation, the bottoming formula is used. Convert the exponent base to 2; use integer shifting to achieve a multiplication approximation, the expression is: ； In the formula, The approximation is, This represents a right shift operation; When performing integer factorization and linear approximation, the exponential term is decomposed into an integer part and a fractional part, represented as: ； In the formula, The integer part For the decimal part, This is a scaling factor used to adjust the range of the decimal part; The decimal part is approximated by a linear function, expressed as: ； In the formula, The integer constant after rounding. ; When performing normalization and integer division operations, the expression is: ； ； In the formula, Output an integer; As molecules; The denominator is ; This is an integer division operation; This is the output bit width; This is the quantization scaling factor for the output integer; = The integer value after the approximate multiplication operation; The output of the Softmax integer operator is .

3. The pure integer quantization method for a visual Transformer model according to claim 1, characterized in that, In the process of approximating GELU as the product of the Sigmoid function and the input to obtain the GELU integer operator... The expression that approximates GELU as the product of the Sigmoid function and the input is: ； In the formula, It is the input of GELU; The input is Operator output at time; The approximation is: It is a sigmoid function operation.

4. The pure integer quantization method for a visual Transformer model according to claim 1, characterized in that, In obtaining the LayerNorm integer operator, the standard deviation is approximated using an integer iteration method to replace the square root operation. The expression for approximating the standard deviation using the integer iteration method is: ； In the formula, It is the first The standard deviation results of the next iteration; It is the first The standard deviation results of the next iteration; For LayerNorm input; for The variance; This represents a right shift operation.

5. A pure integer quantization system for a visual Transformer model, characterized in that, include: The quantization and integration unit is used to integrate linear and nonlinear components based on a selected visual Transformer model to obtain a pure integer quantized model. Specifically, when quantizing nonlinear components, integer approximation algorithms are used to reconstruct the three types of nonlinear operators, Softmax, GELU, and LayerNorm, into integers to obtain reconstructed Softmax, GELU, and LayerNorm integer operators. The visual Transformer model is used for image classification, object detection, or semantic segmentation tasks. The encapsulation and optimization unit is used to encapsulate operators and optimize the computation graph of the pure integer quantization model using the tensor virtual machine compiler framework, so as to obtain an optimized pure integer computation graph. Specifically, when using an integer approximation algorithm to reconstruct Softmax into integers, the Softmax integer operator is obtained by transforming the exponentially-containing Softmax into integer arithmetic operations through exponential stabilization, exponential base-changing and shift approximation, integer decomposition and linear approximation, as well as normalization and integer division operations. When using an integer approximation algorithm to reconstruct GELU into integers, GELU is approximated as the product of the Sigmoid function and the input, thus obtaining the GELU integer operator. When using an integer approximation algorithm to reconstruct LayerNorm into integers, the standard deviation is approximated using an integer iteration method to replace the square root operation, thus obtaining the LayerNorm integer operator. in, Using the Tensor Virtual Machine compiler framework, the pure integer quantization model is encapsulated with operators and the computation graph is optimized to obtain the optimized pure integer computation graph. In this process, the pure integer quantization model is imported into the front end of the Tensor Virtual Machine compiler framework to generate a Relay IR expression, and all tensors are constrained to be of type int8 fixed-point format, carrying fixed-point scaling parameters. The Softmax integer operator, GELU integer operator, and LayerNorm integer operator are encapsulated into Relay-callable integer primitives through the operator registration mechanism in the Tensor Virtual Machine compiler framework, and an integer computation definition is provided for each integer operator. After encapsulation, the corresponding nodes in the original computation graph are replaced with integer primitive calling nodes. Graph-level optimization is performed on the Relay computation graph containing integer primitives. Through operator fusion, constant folding, and memory layout reorganization, the original computation graph is rewritten into a unified integer dataflow computation graph, resulting in the optimized pure integer computation graph. Using the Tensor Virtual Machine compiler framework, the pure integer quantization model is encapsulated into operators and the computation graph is optimized to obtain the optimized pure integer computation graph. Following computation graph optimization, automatic scheduling optimization is also included. In this automatic scheduling optimization process, the intermediate representations corresponding to each fused operator in the unified integer data flow computation graph are used as input. The AutoTVM intelligent scheduling search mechanism in the Tensor Virtual Machine compiler framework automatically generates the optimal execution scheduling strategy for each operator. AutoTVM abstracts the operator execution process into a parameterized execution template determined by a set of scheduling parameters, and uses the following cost function as the optimization objective: ； In the formula, θ represents the set of scheduling parameters; This represents the estimated computational delay of the operator under the current scheduling policy; This represents the estimated storage access latency caused by data memory access and cache access; This indicates the synchronization overhead introduced by pipeline synchronization or parallel control; , , These are coefficient parameters used to balance the weighted relationship between computation cost, memory access cost, and synchronization cost; After obtaining the optimized pure integer computation graph, the process also includes compiling the optimized pure integer computation graph into FPGA executable logic.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the pure integer quantization method for the visual Transformer model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the pure integer quantization method for the visual Transformer model as described in any one of claims 1 to 4.

Images