Progressive lightweight method and device for high-efficiency self-attention model

Abstract

The invention relates to a progressive lightweight method for a high-efficiency self-attention model, which belongs to the technical field of artificial intelligence, and comprises the following steps of: quantizing a weight to a higher bit for intermediate transition, and gradually compressing to a target bit, so that the model can gradually adapt to distribution change caused by quantization in an optimization process, and the weight of the model is reduced. Therefore, the performance loss caused by the quantization error is obviously reduced. According to the mode of combining staged transition and mixed granularity reconstruction, the performance recovery capability of the low-bit quantization model is improved, so that the model can still maintain relatively high accuracy under extremely low bit representation, and has relatively high robustness and practicability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a method and device for progressively lightweighting of a high-performance self-attention model. Background Technology

[0002] In recent years, self-attention, as a key innovation in deep learning, has been widely applied to various artificial intelligence tasks, particularly demonstrating outstanding global modeling capabilities in image classification, object detection, semantic segmentation, and natural language processing. Self-attention model architectures, represented by Transformer, are gradually becoming the mainstream neural network paradigm after Convolutional Neural Networks (CNNs). With the increasing adoption of self-attention models in industrial and smart terminal applications, their deployment on edge computing platforms has become increasingly prominent. For example, multimodal models based on self-attention mechanisms have been widely introduced into the visual perception and natural language interaction modules of smart cockpits to enhance human-computer interaction, environmental understanding, and semantic reasoning capabilities. However, these models typically contain a large number of parameters and computational redundancy, relying on high-performance hardware resources, which severely restricts their application in resource-constrained environments such as mobile devices, embedded terminals, and automotive platforms.

[0003] To address this challenge, model lightweighting has become a key research direction. Quantization, as a mainstream model compression method, significantly reduces the storage and computational overhead of the model by mapping floating-point weights and activation values ​​to low-bit integer representations, while improving inference efficiency. Especially in automotive and edge device scenarios, quantization can effectively reduce energy consumption and latency during inference. By converting floating-point parameters (weights and activations) in the model into low-bit-width integer representations, quantization effectively reduces the model's memory footprint and accelerates the inference process. However, quantization inevitably introduces information loss, typically leading to a decrease in model accuracy, especially in precision-sensitive tasks. To address the accuracy degradation caused by quantization, existing technologies mainly propose two types of methods: Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). QAT simulates quantization noise during training, thus preserving model accuracy to some extent, but requires significant computational resources and time. In contrast, PTQ does not require retraining, can generate quantized models in a shorter time, and has good computational and time efficiency. However, the PTQ method typically fails to achieve the accuracy level of QAT, therefore researchers have proposed various reconstruction optimization strategies to narrow the performance gap between the two. Current research on PTQ reconstruction optimization mainly focuses on the following three types of methods:

[0004] As one of the earliest proposed PTQ reconstruction approaches, the adaptive rounding-based reconstruction method uses layers in the model as the basic unit. During quantization, it introduces additional rounding parameters to adaptively determine the rounding direction of each weight's floating-point value. This method continuously adjusts the rounding parameters through large-scale iterative optimization (typically requiring tens of thousands of iterations) to minimize the difference in single-layer output between the full-precision model and the quantized model, thereby improving the performance of the quantized model. However, this method has certain limitations. On the one hand, its reconstruction optimization process only focuses on the quantization error of a single layer, lacking consideration for the cumulative effect of error propagation between layers; on the other hand, for self-attention models with numerous layers and complex structures, the reconstruction process is extremely time-consuming, resulting in low overall deployment efficiency and limiting its feasibility in practical applications.

[0005] To further alleviate the accuracy loss of self-attention models under low-level quantization conditions, a block-level reconstruction optimization method exists. This method uses blocks as the basic reconstruction unit and employs an error reconstruction mechanism during quantization, primarily targeting adaptive optimization of weight parameters. By directly minimizing the error of the overall block output, it reduces the overall performance degradation caused by quantization. Compared to hierarchical reconstruction methods, block-level reconstruction effectively reduces the complexity of the optimization process. However, this method still has significant limitations: it mainly focuses on minimizing the error of the overall block output, lacking independent optimization of fine-grained errors in individual layers within the block (such as multi-head attention and multilayer perceptrons). This neglect leads to the failure to effectively suppress the accumulation of inter-layer errors within the block, thus limiting the improvement of the final model's accuracy and compression performance to some extent.

[0006] To further improve the adaptability of quantization models in different scenarios, a hybrid granularity reconstruction optimization method exists. This method combines block-level and layer-level reconstruction strategies, performing error control simultaneously at different granularity levels to ensure both quantization accuracy and reconstruction efficiency. Hybrid granularity reconstruction typically utilizes a combination of local and global supervision, performing differentiated optimization on different parts of the model during the reconstruction process, effectively improving model performance under medium-to-high bit quantization conditions. However, existing hybrid granularity reconstruction methods still have certain limitations: most current reconstruction optimizations directly quantize weights and activations to the target bit level for optimization, lacking an adaptation process to quantization, resulting in excessive quantization errors in extremely low bit compression paths. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a high-performance progressive lightweight method and device for self-attention models.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a progressive lightweight method for high-performance self-attention models, comprising the following steps:

[0010] S1: Encapsulate the fully connected layers and convolutional layers in the self-attention model and insert quantizers; the self-attention model is composed of multiple cascaded self-attention blocks, and each self-attention block consists of a multi-head attention mechanism module and a multilayer perceptron module.

[0011] S2: Keep the model weight parameters in full-precision floating-point format, apply quantization processing only to the model activation values, and perform hybrid granularity reconstruction optimization on each self-attention block of the model;

[0012] S3: After the activation quantization parameter stabilizes, keep the activation quantization parameter unchanged, and use a uniform quantizer to quantize the weights in the model from full-precision floating point to high-bit-width quantization transition weights; then re-optimize each self-attention block of the model with mixed granularity reconstruction, and minimize the difference between the quantized model output and the full-precision model output by adjusting the weights, so as to adapt to the distribution changes brought about by the initial quantization of the weights.

[0013] S4: Compress the weight parameters to the target bit level to achieve model representation under low bit conditions. Then, re-optimize each self-attention block of the model with mixed granularity to obtain the final quantized model.

[0014] Furthermore, the self-attention model consists of L cascaded self-attention blocks, targeting the input image. Its dimensions are (h, w), and the number of channels is c. First, the input image is preprocessed to form a patch embedding sequence, which is used as the input to the self-attention block.

[0015] Within each self-attention block, a multi-head self-attention mechanism module and a multilayer perceptron module are sequentially configured. For the l-th self-attention block, its multi-head self-attention mechanism module contains H self-attention heads, and the calculation process for each self-attention head is as follows:

[0016]

[0017] in and This represents the weights and biases of the QKV linear layer. AS i X represents the attention score of attention head i. The output X of the multi-head attention mechanism module. MHA Represented as:

[0018] X MHA = concat([AS1,AS2,...,AS) i ,...,AS H])W o +b o

[0019] Among them W o and b o This represents the weights and biases of the output projection layer in the multi-head attention mechanism module;

[0020] The multilayer perceptron module mainly consists of two linear layers, FC1 and FC2, and its specific calculation process is as follows:

[0021] X′ l =LayerNorm(X MHA +X l-1 )

[0022] X FC1 =GELU(X′) l W FC1 +b FC1 )

[0023] X FC2 =X FC1 W FC2 +b FC2

[0024] Among them W FC1 and b FC1 W represents the weights and biases of the FC1 linear layer. FC2 and b FC2 The output of the l-th self-attention block represents the weights and biases of the FC2 linear layer:

[0025] X l =X′ l +X FC2 .

[0026] Furthermore, step S2, which maintains the model weight parameters in full-precision floating-point format and applies quantization only to the model activation values, performs hybrid-granularity reconstruction optimization on each self-attention block of the model, specifically includes:

[0027] Using the full-precision model as the teacher and the quantization model as the student, forward inference is performed, and the quantization parameters of each module are optimized by minimizing the loss between the output of the quantization model and the output of the full-precision model. The hybrid granularity reconstruction optimization includes: firstly, fine-grained reconstruction is performed on the multi-head attention mechanism module and the multilayer perceptron module in each self-attention block, and the mean square error between the quantization output and the full-precision output is minimized by fine-tuning the weights; after the fine-grained reconstruction is completed, coarse-grained reconstruction is performed on the entire self-attention block.

[0028] Furthermore, the optimization objective for mixed-granularity reconstruction in step S2 is expressed as:

[0029]

[0030] When performing coarse-grained reconstruction and These represent the l-th full-precision self-attention block and the l-th quantized self-attention block, respectively. Each self-attention block consists of a multi-head attention mechanism module and a multilayer perceptron module. During fine-grained reconstruction... This represents the self-attention module or multilayer perceptron module within the l-th full-precision self-attention block. represents the self-attention module or multilayer perceptron module within the l-th quantized self-attention block; W represents the weights of the self-attention block, multi-head attention mechanism module, or multilayer perceptron module in the full-precision model; A represents the activation of the self-attention block, multi-head attention mechanism module, or multilayer perceptron module in the full-precision model; A q W represents the activation of self-attention blocks, multi-head attention mechanisms, or multilayer perceptron modules in the quantization model. f This represents the weights of self-attention blocks, multi-head attention mechanism modules, or multilayer perceptron modules in the quantization model; by optimizing W... f This minimizes the differences between the modules in the quantization model and the modules in the corresponding full-precision model.

[0031] Furthermore, in step S3, the weights W optimized in step S2 are... f* Quantization transition weights are quantized to high bits using a uniform quantizer. Then, the optimization objective for mixed-granularity reconstruction is expressed as:

[0032]

[0033] Furthermore, in step S4, the optimized quantization weights are... The target bit is then quantized again using a uniform quantizer with a quantization weight W. q Then, the optimization objective for mixed-granularity reconstruction is expressed as:

[0034]

[0035] Furthermore, in step S3, the floating-point weights of the model are quantized into 8-bit integer representations using a uniform quantizer; in step S4, the weights are quantized from 8 bits to a target bit width of 3 bits using a uniform quantizer.

[0036] In a second aspect, the present invention provides an electronic device, including a memory and a processor;

[0037] The memory is used to store computer programs;

[0038] The processor is configured to implement the high-performance self-attention model progressive lightweight method as described above when executing the computer program.

[0039] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the high-performance progressive lightweight method for self-attention models as described above.

[0040] The beneficial effects of this invention are as follows:

[0041] 1. Employing a hybrid granularity progressive quantization strategy, this invention progressively optimizes model parameters during the quantization process, minimizing quantization error through fine-grained and coarse-grained reconstruction optimization. Particularly in the low-bit quantization stage, by repeatedly fine-tuning the weights, the model's performance is restored to the maximum extent, ensuring that the quantized model maintains high performance even under extremely low bit conditions.

[0042] 2. In the reconstruction strategy, this invention introduces a progressive quantization strategy for weight quantization. First, the weights are quantized to a high bit level to adapt to the quantization process, and then gradually compressed to the target bit level. Combined with mixed granularity reconstruction optimization, this effectively alleviates the feature distortion and sudden drop in model performance caused by traditional one-step quantization, and improves the stability of the quantization process and the robustness of the model.

[0043] 3. The method of the present invention is simple and easy to implement, while saving computational overhead and greatly improving performance. The performance of the method of the present invention exceeds that of various mainstream post-training quantization methods, especially when the bit depth is lower.

[0044] 4. This invention also features high energy efficiency, enabling efficient model deployment and inference with limited computing resources, further enhancing the applicability and practicality of the model on resource-constrained platforms such as edge devices and mobile terminals.

[0045] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart of the progressive lightweight method for high-performance self-attention models described in Embodiment 2 of the present invention. Detailed Implementation

[0048] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0049] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0050] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0051] Example 1:

[0052] This invention provides a progressively lightweight method for high-performance self-attention models. This embodiment takes a language self-attention model in a smart cockpit as an example, and the specific steps are as follows:

[0053] The fully connected layers and convolutional layers in the model are encapsulated and the corresponding quantizers are inserted;

[0054] In Phase 1, the model weights are kept in full-precision floating-point format. Quantization is applied only to the model activation values. Each block of the model and its key modules are recursively traversed, undergoing hybrid-granularity reconstruction optimization. During reconstruction, the full-precision model acts as the teacher, and the quantized model as the student, performing forward inference. By minimizing the loss between the quantized and full-precision model outputs, the quantization parameters of each module are optimized, ensuring that the output features after activation quantization are as close as possible to the output features in the unquantized state, thereby reducing the precision loss caused by activation quantization. In this phase, fine-grained reconstruction is first performed on the self-attention module and the multilayer perceptron module, optimizing the quantization parameters of these key modules. After fine-grained reconstruction, coarse-grained reconstruction is performed on the entire block. In this way, the hybrid-granularity reconstruction process is executed block by block, ensuring that the impact of quantization error on model performance is minimized during the activation quantization phase.

[0055] In Phase Two, after the activation quantization parameters stabilize, weight quantization initialization begins. The weights in the model are quantized from full-precision floating-point to higher-bit-width quantization using a uniform quantizer. After weight quantization, both the activation and weights are in a quantized state (activation is target-bit quantization, weights are high-bit quantization). Under these conditions, the entire model is re-traversed, and the aforementioned mixed-granularity reconstruction optimization is performed. By adjusting the weights, the difference between the quantized model output and the full-precision model output is minimized, adapting to the distribution changes caused by the initial weight quantization, thus laying the foundation for the final low-bit quantization stage.

[0056] Phase Three follows the optimization completed in Phase Two's high-bit weight quantization stage. The weight parameters are further compressed to the target bit level to achieve model representation under low-bit conditions. Activation quantization maintains the original settings, and after weight quantization is reduced from high bit level to the target bit level, reconstruction optimization is performed to restore model performance. This reconstruction stage still employs hybrid granularity reconstruction optimization, aiming to restore the output characteristics of the model after extremely low-bit quantization as much as possible. By repeatedly fine-tuning the weights, the impact of quantization errors on model inference performance is minimized, allowing the quantized language self-attention model to maintain high performance even under extremely low-bit conditions.

[0057] Through the above reconstruction process, the present invention can effectively reduce quantization error during quantization and ensure that the quantized language self-attention model can still maintain high performance under extremely low bit conditions.

[0058] In an optional embodiment, the mixed-granularity reconstruction optimization objective of stage one can be expressed as:

[0059]

[0060] The weight W after Phase 1 optimization f* Use a uniform quantizer to quantize to a higher bit quantization transition weight. Subsequently, the optimization objective for the mixed-granularity reconstruction in stage two can be expressed as:

[0061]

[0062] The quantized weights after Phase 2 optimization The target bit is then quantized again using a uniform quantizer with a quantization weight W. q Subsequently, the optimization objective for the mixed-granularity reconstruction in stage three can be expressed as:

[0063]

[0064] Example 2:

[0065] This embodiment takes the visual self-attention model in an autonomous driving scenario as an example, and provides a quantization process on the visual self-attention model ViT-Base. The quantization parameters are set with weight quantization of 3 bits, transition weight quantization of 8 bits, and activation quantization of 3 bits, and a specific implementation method is shown on the ImageNet dataset.

[0066] Please see Figure 1 The quantification method includes the following steps:

[0067] S1: Quantization Model Construction; Select the target visual self-attention model ViT-Base, encapsulate the fully connected layers and convolutional layers in the model according to the quantization parameters, and insert the provided quantizer. The overall architecture of the visual self-attention model consists of L cascaded self-attention blocks. For the input image... Its dimensions are (h, w), and the number of channels is c. To adapt the input to the self-attention model structure, the input image is first preprocessed, specifically including: the original image I is divided into n small patches. Each patch has a resolution of h. p ×w p Then, a linear transformation is performed on each patch, projecting it into a d-dimensional vector. The corresponding positional encoding is added to each embedded patch vector to form the final patch embedding sequence. This processed embedded patch sequence serves as the input to the self-attention block. (For the sake of simplicity, the process of introducing classification embeddings is omitted here.)

[0068] Within each self-attention block, a multi-head self-attention mechanism module and a multilayer perceptron module are sequentially configured. Specifically, for the l-th self-attention block, its multi-head self-attention mechanism module contains H self-attention heads. The calculation process for each self-attention head is as follows:

[0069]

[0070] in and This represents the weights and biases of the QKV linear layer. AS i X represents the attention score of attention head i. The output X of the multi-head attention mechanism module. MHA Represented as:

[0071] X MHA = concat([AS1,AS2,...,AS) i ,...,AS H ])W o +b o

[0072] Among them W oand b o This represents the weights and biases of the output projection layer in the multi-head attention mechanism module.

[0073] The multilayer perceptron module mainly consists of two linear layers (denoted as FC1 and FC2, respectively), and its specific calculation process is as follows:

[0074] X′ l =LayerNorm(X MHA +X l-1 )

[0075] X FC1 =GELU(X′) l W FC1 +b FC1 )

[0076] X FC2 =X FC1 W FC2 +b FC2

[0077] Among them W FC1 and b FC1 W represents the weights and biases of the FC1 linear layer. FC2 and b FC2 This represents the weights and biases of the FC2 linear layer. The output of the l-th self-attention block is given by the following formula:

[0078] X l =X′ l +X FC2

[0079] S2: In the first stage, quantization is performed only on the activation components, while the weights remain in full-precision floating-point state. The goal of this stage is to suppress the quantization error of the activation values ​​to minimize the impact of the input distribution on the model performance in subsequent stages. Specifically, a hybrid granularity optimization reconstruction approach is used. First, at a fine-grained level, reconstruction optimization is performed on each multi-head attention mechanism module and multilayer perceptron module within each self-attention block. By fine-tuning the weights, the mean square error between the quantized output and the full-precision output is minimized. After completing the fine-grained optimization, a coarse-grained reconstruction is performed on the entire module to enhance the consistency of block-level outputs. In this stage, the full-precision model acts as the teacher model, and the quantized model acts as the student model. The parameters of the quantized model are updated through backpropagation to minimize the gap between the quantized model and the full-precision model.

[0080] S3: The second stage introduces weight quantization. In this stage, the activation quantization parameters remain unchanged, and the model's floating-point weights are quantized into higher-width 8-bit integer representations using a uniform quantizer, serving as transition weights. Since changes in weight distribution can cause performance fluctuations, this stage still requires a complete hybrid granularity reconstruction process. Unlike stage one, the current optimization focus is on adapting to the model's distribution after initial weight quantization. Using the same hybrid reconstruction method, output features are compared, and the weights of the quantized model are fine-tuned to reduce quantization errors, ensuring the model achieves relatively stable performance under high-bit quantization, providing good initial conditions for the final ultra-low bit compression.

[0081] S4: In the third stage, based on the above optimizations, the weights are further compressed from 8 bits to the target bit width of 3 bits to achieve the final model quantization representation. In this stage, the activation quantization configuration remains unchanged, and only the quantized model weights are quantized to the target bit width. Subsequently, a hybrid granularity reconstruction optimization of the quantized model is adopted. Specifically, the weights optimized in stage two are used as initialization. The quantizer requantizes the weights using the target bit configuration, and the hybrid granularity reconstruction mechanism is still used to reduce quantization errors, thereby allowing the model to achieve good performance even with low bit width.

[0082] In the progressive reconstruction optimization strategy of this invention, quantization transition weights refer to quantizing the original full-precision floating-point weights in the model into an intermediate quantized weight form with a higher bit width (e.g., 8 bits) before quantizing the extremely low-bit weights. This intermediate weight is not used as the output weight of the final deployed model, but rather serves as a "transitional state" between the full-precision weights and the target low-bit weights during the reconstruction optimization process. The core purpose of introducing quantization transition weights is to construct a progressive adaptation path "from full precision to low bit width." By first quantizing the weights to a higher bit width such as 8 bits, the model can perform reconstruction optimization with minimal precision loss, completing preliminary quantization adaptation and feature alignment. Then, the weights are further quantized to the target bit width, and reconstruction is performed on this basis, thereby more effectively suppressing the precision degradation caused by low bits. This mechanism can significantly improve the stability and recoverability of the quantization process. Compared with the traditional method of directly quantizing floating-point weights to the target bit width in one step, the quantization transition weight strategy can reduce quantization errors, enabling the final quantized model to still have high precision performance under extremely low bit conditions.

[0083] All methods in this invention are implemented using the deep learning framework PyTorch, and uniform quantization is applied to all weights and activations. The calibration dataset used in the experimental setup consists of 1024 images from the ImageNet dataset. During the reconstruction process, all self-attention blocks in the visual self-attention model, as well as the multi-head attention mechanism module and the multilayer perceptron module therein, are iterated 800 times.

[0084] The quantization models include two advanced visual self-attention models: the ViT series and the Deit series. These models were tested using both 4-bit and 3-bit quantization.

[0085] This invention can be directly applied to post-training quantization of visual self-attention models. By rapidly reconstructing, optimizing, and compressing the visual self-attention model, it improves the model's accuracy and efficiency in processing images.

[0086] Tables 1 and 2 compare the quantitative results of existing state-of-the-art quantization methods and the method of this invention on the same dataset. It can be seen that, for visual self-attention models with different bit widths, the method of this invention outperforms all the compared methods. Compared with existing state-of-the-art quantization methods, the method of this invention achieves a significant performance improvement at low bit widths.

[0087] Table 1

[0088]

[0089]

[0090] Table 2

[0091]

[0092] Tables 1 and 2 present the quantitative results of current leading visual self-attention models on ImageNet. Table 2 shows the performance of the proposed method on ViT series models under the same settings. In a 3-bit configuration, traditional methods such as PTQ4ViT, FQ-ViT, and RepQ-ViT achieve accuracy close to zero or even less than 1%. While optimized methods such as QDrop and PD-Quant offer some improvement, their accuracy remains low. The progressive quantization method proposed in this invention achieves 58.95% accuracy on ViT-S and 74.85% on ViT-B, significantly outperforming all other methods, demonstrating the significant performance recovery capability of the proposed method under extremely low bit compression conditions. In a 4-bit configuration, the proposed method still achieves the best accuracy in both models, further validating its effectiveness at lower bit depths. Table 2 also shows the performance of the proposed method on DeiT series models under the same settings. Experimental results show that, under 3-bit configuration, the proposed method achieves Top-1 accuracy of 64.82% and 74.62% on DeiT-S and DeiT-B models, respectively, significantly outperforming other optimized methods such as I&S-ViT (55.78% and 73.30%, respectively) and QDrop (22.67% and 24.37%), demonstrating the quantization advantage of this invention in various visual self-attention models. Under 4-bit configuration, the proposed method remains stable on DeiT-S and DeiT-B models, with accuracies of 76.08% and 80.48%, respectively, both exceeding the current best methods ADFQ-ViT and I&S-ViT.

[0093] In summary, the data in Tables 1 and 2 fully demonstrate that the method of the present invention is significantly superior to existing methods in the post-training quantization scenario of Transformer models, not only under extremely low bit conditions but also maintaining high competitiveness under low bit quantization, and has extremely strong practical value.

[0094] The technical solutions provided in this application embodiment have at least the following technical effects or advantages: This invention proposes a progressive reconstruction optimization strategy, addressing the low-bit precision degradation problem that easily occurs in the post-training quantization process of visual self-attention models. A progressive weight quantization process is introduced into the model reconstruction optimization process. By first quantizing the weights to a higher bit level for an intermediate transition, and then gradually compressing them to the target bit level, the model can gradually adapt to the distribution changes brought about by quantization during the optimization process, thereby significantly reducing the performance loss caused by quantization errors. This combination of staged transition and mixed-granularity reconstruction improves the performance recovery capability of low-bit quantization models, enabling the model to maintain high accuracy even with extremely low bit (e.g., 3-bit) representation, exhibiting stronger robustness and practicality. Although specific embodiments of this invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of this invention should be covered within the scope of protection of the claims of this invention.

[0095] Example 3:

[0096] An electronic device, comprising a memory and a processor;

[0097] The memory is used to store computer programs;

[0098] The processor is configured to implement the method described in any of Embodiments 1-2 when executing the computer program.

[0099] Example 4:

[0100] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of Embodiments 1-2.

[0101] Example 5:

[0102] A computer program product includes a computer program that, when executed by a processor, implements the method described in any of Examples 1-2.

[0103] In the above embodiments, the reference to "this embodiment" in the specification indicates that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple appearances of "this embodiment" do not necessarily refer to the same embodiment.

[0104] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0105] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0106] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0107] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0108] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0109] This invention can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0110] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A progressively lightweight method for high-performance self-attention models, characterized in that: Includes the following steps: S1: Encapsulate the fully connected layers and convolutional layers in the self-attention model and insert quantizers; the self-attention model is composed of multiple cascaded self-attention blocks, and each self-attention block consists of a multi-head attention mechanism module and a multilayer perceptron module. S2: Keep the model weight parameters in full-precision floating-point format, apply quantization processing only to the model activation values, and perform hybrid granularity reconstruction optimization on each self-attention block of the model; S3: After the activation quantization parameter stabilizes, keep the activation quantization parameter unchanged, and use a uniform quantizer to quantize the weights in the model from full-precision floating point to high-bit-width quantization transition weights. Then, the model's self-attention blocks are reconstructed and optimized at a mixed granularity. By adjusting the weights, the difference between the quantized model output and the full-precision model output is minimized, adapting to the distribution changes caused by the initial weight quantization. S4: Compress the weight parameters to the target bit level to achieve model representation under low bit conditions. Then, re-optimize each self-attention block of the model with mixed granularity to obtain the final quantized model.

2. The progressive lightweight method for high-performance self-attention models according to claim 1, characterized in that: The self-attention model consists of L cascaded self-attention blocks, and is designed for the input image. Its dimensions are (h, w), and the number of channels is c. First, the input image is preprocessed to form a patch embedding sequence, which is used as the input to the self-attention block. Within each self-attention block, a multi-head self-attention mechanism module and a multilayer perceptron module are sequentially configured. For the l-th self-attention block, its multi-head self-attention mechanism module contains H self-attention heads, and the calculation process for each self-attention head is as follows: in and This represents the weights and biases of the QKV linear layer. AS i The attention score represents the attention score of attention head i. The output X of the multi-head attention mechanism module MHA Represented as: X MHA =concat([AS1,AS2,...,AS i ,...,AS H ])W o +b o Among them W o and b o This represents the weights and biases of the output projection layer in the multi-head attention mechanism module; The multilayer perceptron module mainly consists of two linear layers, FC1 and FC2, and its specific calculation process is as follows: X l '=LayerNorm(X MHA +X l-1 ) X FC1 =GEL(X l 'W FC1 +b FC1 ) X FC2 =X FC1 W FC2 +b FC2 Among them W FC1 and b FC1 W represents the weights and biases of the FC1 linear layer. FC2 and b FC2 The output of the l-th self-attention block represents the weights and biases of the FC2 linear layer: X l =X l '+X FC2 。 3. The progressive lightweight method for high-performance self-attention models according to claim 1, characterized in that: Step S2, which maintains the model weight parameters in full-precision floating-point format and applies quantization only to the model activation values, performs hybrid-granularity reconstruction optimization on each self-attention block of the model, specifically includes: Using the full-precision model as the teacher and the quantization model as the student, forward inference is performed, and the quantization parameters of each module are optimized by minimizing the loss between the output of the quantization model and the output of the full-precision model. The hybrid granularity reconstruction optimization includes: firstly, fine-grained reconstruction is performed on the multi-head attention mechanism module and the multilayer perceptron module in each self-attention block, and the mean square error between the quantization output and the full-precision output is minimized by fine-tuning the weights; after the fine-grained reconstruction is completed, coarse-grained reconstruction is performed on the entire self-attention block.

4. The progressive lightweight method for high-performance self-attention models according to claim 3, characterized in that: The optimization objective for mixed-granularity reconstruction in step S2 is expressed as: When performing coarse-grained reconstruction and These represent the l-th full-precision self-attention block and the l-th quantized self-attention block, respectively. Each self-attention block consists of a multi-head attention mechanism module and a multilayer perceptron module. During fine-grained reconstruction... This represents the self-attention module or multilayer perceptron module within the l-th full-precision self-attention block. represents the self-attention module or multilayer perceptron module within the l-th quantized self-attention block; W represents the weights of the self-attention block, multi-head attention mechanism module, or multilayer perceptron module in the full-precision model; A represents the activation of the self-attention block, multi-head attention mechanism module, or multilayer perceptron module in the full-precision model; A q W represents the activation of self-attention blocks, multi-head attention mechanisms, or multilayer perceptron modules in the quantization model. f This represents the weights of self-attention blocks, multi-head attention mechanism modules, or multilayer perceptron modules in the quantization model; by optimizing W... f This minimizes the differences between the modules in the quantization model and the modules in the corresponding full-precision model.

5. The progressive lightweight method for high-performance self-attention models according to claim 4, characterized in that: In step S3, the weights W optimized in step S2 are... f* Quantization transition weights are quantized to high bits using a uniform quantizer. Then, the optimization objective for mixed-granularity reconstruction is expressed as:

6. The progressive lightweight method for high-performance self-attention models according to claim 5, characterized in that: In step S4, the optimized quantization weights are... The target bit is then quantized again using a uniform quantizer with a quantization weight W. q Then, the optimization objective for mixed-granularity reconstruction is expressed as:

7. The progressive lightweight method for high-performance self-attention models according to claim 1, characterized in that: In step S3, the floating-point weights of the model are quantized into 8-bit integer representations using a uniform quantizer; in step S4, the weights are quantized from 8 bits to the target bit width of 3 bits using a uniform quantizer.

8. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the high-performance self-attention model progressive lightweighting method as described in any one of claims 1-7 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the high-performance self-attention model progressive lightweighting method as described in any one of claims 1-7.

10. A computer program product, characterized in that: Includes a computer program that, when executed by a processor, implements the high-performance self-attention model progressive lightweighting method as described in any one of claims 1-7.

Images