A method for reducing gradient bias in multi-task network quantization training process

Abstract

The application discloses a method for reducing gradient deviation in multi-task network quantization training process, and relates to the technical field of image processing, comprising the following steps: in the process of quantization training of the multi-task network with images as input, image shallow features and image deep features that need to be fused are obtained; the image shallow features and the image deep features are multiplied by learnable weights respectively; two multiplication results are taken as inputs of corresponding feature fusion layers, and then fused image features are obtained; the fused image features are normalized on all channels at each spatial position to obtain normalized image features; and the normalized image features are taken as inputs of subsequent structures of the feature fusion layers of the multi-task network. The method can optimize the balanced contribution of shallow branches and deep branches in the multi-task network in the quantization training process, reduce the gradient deviation in the multi-task network quantization training process, and improve the precision of the multi-task network after quantization training.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for reducing gradient bias during the quantization training process of multi-task networks. Background Technology

[0002] In recent years, quantization has become a key technique in neural network model compression. By converting the representation of network weights and activations from high-precision floating-point format to low-bit integer form, quantization can effectively reduce memory overhead and computational latency during deployment. Current quantization methods mainly include Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ does not require retraining and can complete model compression using only a small calibration set; while QAT inserts pseudo-quantization operations during training and uses a straight-through estimator (STE) to approximate the gradient, significantly improving accuracy preservation while increasing training overhead.Classic PTQ methods such as AdaRound (Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. InProceedings of the 37th International Conference on Machine Learning (ICML'20), Vol. 119. JMLR.org, Article 667, 7197–7206), BRECQ (Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu, “BRECQ: Pushing thelimit of post-training quantization by block reconstruction,” in Proc. Int.Conf. Learn. Representations (ICLR), 2021) and QDrop (X. Wei, R. Gong, Y. Li, X. Liu, and F. Yu, “QDrop: Randomly dropping The paper "Quantization for extremely low-bit post-training quantization," in Proc. Int. Conf. Learn. Representations (ICLR), 2022, has shown strong performance under 8-bit quantization. However, at even lower bit widths, the QAT method exhibits a significant advantage.Representative state-of-the-art QAT methods include: PACT (Choi, Jungwook et al. “PACT: Parameterized Clipping Activation for Quantized Neural Networks.” ArXiv abs / 1805.06085 (2018): n. pag), which reduces quantization error by suppressing outliers in activation values; LSQ (SKEsser, JL McKinstry, D. Bablani, R. Appuswamy, and DS Modha, “LEARNEDSTEP SIZE QUANTIZATION,” in Proc. Int. Conf. Learn. Representations (ICLR), 2020), which jointly optimizes weights and scaling factors during training; and LSQ+ (Y. Bhalgat, J. Lee, M. Nagel, T. Blankevoort and N. Kwak, "LSQ+: Improving low-bit quantization through learnable offsets and better initialization," 2020), which further introduces learnable offsets and improved initialization strategies on the basis of LSQ; and N2UQ (Liu Z, Cheng KT, Huang D, et al. Nonuniform-to-uniform quantization: Towards accurate quantization via generalized straight-through estimation[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2022: 4942-4952), which enhances the flexibility of feature representation by learning input thresholds while maintaining a hardware-friendly uniform quantization method.

[0003] These methods achieve excellent accuracy performance for quantization of single image classification task models at low bit widths (≤4 bits), and even surpass their full-precision models at low bit widths. For example, the N2UQ method, as the current state-of-the-art quantization-aware training method, achieves better accuracy performance than its full-precision model in image classification tasks with a 4-bit configuration. However, in the context of multi-task learning (simultaneously optimizing multiple tasks within a single network framework, such as object detection and recognition, image segmentation and recognition), existing technologies cannot yet achieve model quantization with low accuracy loss, especially for architectures that perform classification and regression tasks simultaneously. Taking the YOLOv5 object detection architecture, which is widely used in both academic research and industrial applications, as an example, although recent research shows that the Q-YOLO method (Wang, M. et al. (2023). Q-YOLO: Efficient Inference for Real-Time Object Detection. In: Lu, H., Blumenstein, M., Cho, SB., Liu, CL., Yagi, Y., Kamiya, T. (eds) Pattern Recognition. ACPR2023. Lecture Notes in Computer Science, vol 14408. Springer, Cham) can quantize the YOLOv5 model with 8-bit precision and maintain near-full-precision performance, as shown in Table 1, when using a low-bit configuration, even when applying state-of-the-art methods to multi-task network scenarios, they still experience a significant decrease in accuracy.

[0004] Table 1: Performance of state-of-the-art quantization methods on the multi-task architecture YOLOv5 with mean average precision (mAP). Experimental results are based on the COCO dataset, where the Q-YOLO data comes from the original source, and the other data are reproduced by ourselves.

[0005]

[0006] In modern object detectors, classification and regression branches are typically set simultaneously. However, because these branches are optimized independently, inconsistencies in prediction results can easily arise: some bounding boxes, despite high classification confidence, exhibit significant localization errors; while others, although accurately localized, are ignored due to low classification confidence. This discrepancy is further exacerbated under quantized-aware training, especially with low-bit quantization, where performance degradation is more severe. Some studies have proposed the Harmonious Quantization for Object Detection (HQOD) framework, which includes two key loss functions: Task-Correlated Loss (TCor) and Harmonious IoU Loss (HIoU).

[0007] Task-related loss is defined by a "task relevance index". ,in To score by category, The IOU score is used. When the difference between the classification and localization scores is large, If the value is lower, the sample is given greater weight in the optimization process, encouraging the network to focus on optimizing samples where "classification and localization are inconsistent". The formula is:

[0008]

[0009] Next, the Harmonious IoU Loss (HIoU) introduces adaptive weighting to address the bias in IoU levels (too many low-IoU samples dominating the optimization):

[0010]

[0011] in, This increases the weight of high IoU samples and suppresses the excessive dominance of low IoU samples in the regression branch.

[0012] Therefore, the overall loss function is:

[0013]

[0014] in, The standard detection loss is (classification + regression). =1.5.

[0015] HQOD, as a new loss function, improves the accuracy of quantization models under both 2-bit and 4-bit conditions.

[0016] However, while HQOD alleviates the performance degradation of quantization models to some extent, its performance remains unsatisfactory on some multi-task architectures. For example, on YOLOX-tiny (L. Huang et al., "HQOD: Harmonious Quantization for Object Detection," 2024 IEEE International Conference on Multimedia and Expo (ICME), Niagara Falls, ON, Canada, 2024, pp. 1-6, doi:10.1109 / ICME57554.2024.10687589.), mAP still decreases by 7.1%, indicating that its effect on improving quantization accuracy in complex multi-task scenarios is limited, and its overall performance is insufficient. HQOD lacks a systematic analysis based on the mathematical essence of multi-task quantization, only proposing empirical indicators and loss functions for improvement. This makes the method highly dependent on task specificity, making it difficult to effectively extend to other multi-task combinations, thus resulting in poor transferability. HQOD also requires modifications to the training process. This strong coupling with the training process means that the method is not an out-of-the-box, plug-in approach that is completely decoupled from existing training, limiting its flexible application in different frameworks and existing training.

[0017] Some studies have also attempted to improve the performance of quantization models by modifying the original network structure, such as designing a backbone structure that is more suitable for quantization, or adapting the existing structure to a quantization scheme and replacing the original CSP backbone network of YOLO.

[0018] Some scholars have pointed out that in PTQ fully quantized detectors (Y. Ding, W. Feng, C. Chen, J. Guo and X. Liu, "Reg-PTQ: Regression-specialized Post-training Quantization for Fully Quantized Object Detector," 2024 IEEE / CVF Conference on ComputerVision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 16174-16184), the regression head is more sensitive to quantization noise than the classification head. Therefore, they proposed a regression-specific PTQ method, which achieved excellent performance in detection networks with ResNet as the backbone. However, this method mainly optimizes for the ResNet architecture and fails to cover a series of networks with stronger multi-task coupling, such as YOLO / CSP. Some scholars have found that the architectures commonly used in modern multi-task networks suffer from significant accuracy degradation during quantization (Xiangxiang Chu, Liang Li, and Bo Zhang. 2024. Make RepVGG greater again: a quantization-aware approach. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence (AAAI'24 / IAAI'24 / EAAI'24), Vol. 38. AAAI Press, Article 1297, 11624–11632). Therefore, they proposed QARepVGG, a novel quantization-friendly backbone network. This method significantly alleviates, or even almost eliminates, the performance degradation caused by quantization by modifying the branch design during the training phase to make the weight and activation distribution more stable.

[0019] However, such modifications often disrupt the original design of multi-task networks, leading to a decline in baseline performance and weakening the original model's performance at full accuracy. Furthermore, their applicability and generalization ability are significantly limited across different tasks or scenarios, thus exhibiting considerable limitations and uncertainties in practical applications.

[0020] To mitigate the oscillation of weights and scale factors around the quantization threshold caused by the STE approximation in the pass-through estimator, some researchers have proposed two improvements to the LSQ method based on modern single-stage YOLO detectors (Gupta, Kartik and Akshay Asthana. “Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks.” 2024 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV) (2023): 2440-2449). These improvements are independent of the network architecture but closely related to the training process: First, EMA (Exponential Moving Average) smoothing, which uses an exponential moving average to apply to the latent weights and quantization scale to suppress iterative oscillations; second, QC (Quantization Correction) post-correction, which freezes the network parameters after QAT and only applies affine correction to the pre-activation of each convolutional output. This approach directly addresses the accuracy degradation problem caused by training oscillations in the YOLO architecture under low-bit QAT, and can serve as an external module for existing QATs, effectively reducing oscillations and improving mAP.

[0021] However, the design and validation of the above methods are limited to the YOLO series. They are effective for YOLO detection and segmentation tasks, but difficult to generalize to other types of multi-task networks, and therefore cannot represent the general problems of multi-task quantization. Moreover, the starting point of the above methods is mainly to alleviate the oscillation phenomenon in quantization training, lacking theoretical modeling of inter-task conflicts, and the ability to provide universal explanations for heterogeneous multi-task heads and cross-architecture transfer is still insufficient.

[0022] To reveal the underlying mechanisms of the above problems, we take YOLOv5, a commonly used multi-task network in scientific research, as an example. Figure 2As shown, YOLOv5 extracts features through the CSP backbone network, fuses these features via the PANet neck structure, and finally completes detection using the detection head. Unlike traditional classification networks that rely solely on the final fully connected layer for decision-making, multi-task networks employ feature fusion techniques to coordinate the needs of different tasks. This is a key design feature of multi-task learning compared to single-task learning. This is because shallow features in the network are better at capturing fine spatial features and are more suitable for regression tasks; while deep features can encode global context and high-level semantics, making them more suitable for classification tasks.

[0023] In quantization-aware training, each quantized layer introduces discretization noise, i.e., quantization error. As network depth increases, these errors accumulate, leading to a cascading effect. These quantization errors are crucial for gradient updates during backpropagation. Therefore, in multi-task architectures, the quantization process can easily introduce significant gradient imbalances at each feature fusion point (the Concat layer in YOLO). Shallow branches suffer from insufficient simulated noise, limiting gradient flow; while deeper branches are hampered by accumulated quantization noise, hindering the optimization process and causing the model to shift its optimization for different features. This was addressed by training a YOLOv5 network on the VOC dataset and recording its feature fusion layers (such as...). Figure 2 As shown, this represents the average feature gradient of different branches in the first Concat layer. Figure 3 As shown, fine-grained local features from shallow layers are crucial for bounding box regression; conversely, high-level semantic features from deeper layers are more important for classification tasks. We observed that the YOLOv5 network tends to optimize deeper semantic branches during quantization training, while ignoring shallow local branches, leading to biases in gradient updates across tasks.

[0024] In the quantization perception training process, each layer introduces quantization error. Let... Indicates the first Full-precision output of the layer, Let represent its quantized value. Then, the quantization error can be defined as: And because of the first The layer's input is also quantized; we have This results in the following output after perturbation: + ,in Indicates the first Layer transformations (e.g., convolution and activation functions). This represents the new quantization noise. Assume... If the value is small, then the first-order approximation is: ,in yes Regarding its input Jacobian matrix, the propagation relationship of the total error is:

[0025]

[0026] This indicates that errors in shallow layers propagate backward through the Jacobian transform and are combined with noise from other layers. This accumulation eventually leads to the inevitable accumulation of errors.

[0027] In multi-task learning scenarios, classification and regression tasks are often involved simultaneously. Cross-entropy loss (widely used in classification tasks) has been successfully applied to the training of quantized neural networks and has proven robustness guarantees. Therefore, our focus is on regression tasks that require high pixel-level accuracy, such as CIoU loss commonly used in object detection.

[0028] To fully utilize shallow and deep feature information, models typically fuse shallow outputs. With deep output For example, fusion can be achieved by splicing: .

[0029] Let the target center offset predicted by the model be... The actual value is denoted as The prediction consists of a linear combination of shallow and deep features:

[0030]

[0031] in, and These represent the linear weights of shallow and deep features, respectively; and This indicates full-precision feature output; This indicates the corresponding quantization error.

[0032] Since CIoU loss includes the center prediction error The related squared terms can be simplified to the following form:

[0033]

[0034] Expanding the error term yields:

[0035]

[0036] like Figure 3 The analysis shows that we can identify which features the network prioritizes during training by observing the gradient flow. In the quantization fine-tuning phase, training begins with a model that has already fully converged at full precision; therefore, the gradients primarily come from... Because quantization noise accumulates continuously between layers, the perturbation term of deep features... The contribution to the final error is greater than that of the terms corresponding to shallow features. .

[0037] To minimize loss, the network amplifies responses to error-driven features and obtains stronger gradient signals from deeper layers during backpropagation, thus prioritizing the updating of deeper features. This ignores crucial shallow, fine-grained information. This effect can be quantified using the following gradient inequality:

[0038]

[0039] This also reveals another important issue: in quantization training, relying solely on loss supervision based on numerical perturbations is insufficient to reflect the model's decision-making ability, because the model's final decision is based on feature representations, rather than simple numerical biases.

[0040] In summary, due to the strong limitations of existing technologies, such as dependence on specific network structures or applicability only to specific tasks, they lack good versatility and their accuracy performance still differs greatly from full accuracy. Summary of the Invention

[0041] To address the aforementioned shortcomings in existing technologies, this invention provides a method for reducing gradient bias during the quantization training process of multi-task networks. This solves the problem that existing quantization training methods for multi-task networks rely on specific network structures or are only applicable to specific tasks, resulting in poor versatility and a significant difference in accuracy performance compared to full precision.

[0042] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0043] A method for reducing gradient bias during the quantization training process of multi-task networks is provided, comprising the following steps:

[0044] In the process of quantizing and training a multi-task network with images as input, the shallow and deep features of the image that need to be fused in the feature fusion layer of the multi-task network are obtained.

[0045] The shallow image features are multiplied by the first learnable weight, and the deep image features are multiplied by the second learnable weight to optimize the balanced contribution of the shallow and deep branches in the multi-task network quantization training process.

[0046] The results of the two multiplications are used as input to the corresponding feature fusion layer to obtain the fused image features;

[0047] The fused image features are normalized for all channels at each spatial location to obtain normalized image features;

[0048] Normalized image features are used as input to the subsequent structure of the feature fusion layer of the multi-task network to reduce gradient bias during the quantization training process and improve the accuracy of the multi-task network after quantization training.

[0049] Furthermore, the expression for multiplying the shallow image features by the first learnable weight and the deep image features by the second learnable weight is as follows:

[0050]

[0051]

[0052] in shallow features of the image With the first learnable weight The result of multiplication; For deep image features With the second learnable weight The result of multiplication.

[0053] Furthermore, and The sum is 1.

[0054] Furthermore, the specific methods for normalizing all channels at each spatial location of the fused image features to obtain normalized image features include:

[0055] Calculate the mean of the fused image features. and standard deviation Its expression is:

[0056]

[0057]

[0058] in and These represent the number of channels for shallow and deep image features, respectively. For channel indexing; For the fused image features in the first... Image features under a single channel;

[0059] The fused image features are normalized for each channel using the following expression:

[0060]

[0061] in For the normalized first Image features under a single channel;

[0062] By iterating through each channel, the normalized image features are obtained.

[0063] Furthermore, the multi-task network is a network that includes classification and regression tasks, used for object detection or image segmentation; where object detection includes image classification and localization.

[0064] Furthermore, the following operations are also included in the multi-task network quantization training process:

[0065] The pre-trained full-precision model is used as the teacher network, and the multi-task network to be quantized and trained is used as the quantization model.

[0066] The same image is used as input to both the teacher network and the quantization model. The attention distribution is extracted from the teacher network and the quantization model respectively through the SimAM mechanism, and the mean and variance of each channel are calculated.

[0067] The mean and variance obtained based on the SimAM mechanism define an inverse energy score for each spatial location of the input image;

[0068] The inverse energy score is mapped through the sigmoid function to obtain the attention weights between 0 and 1 at each spatial location, thereby obtaining the attention distribution corresponding to the teacher network and the attention distribution corresponding to the quantization model;

[0069] The original image features of spatial location are weighted with their corresponding attention weights to obtain the enhanced image features, i.e., the enhanced image features A corresponding to the teacher network. f Enhanced image features A corresponding to the quantization model q ;

[0070] The attention distributions corresponding to the teacher network and the quantization model are normalized to probability distributions P and Q using Softmax, respectively. The difference between probability distributions P and Q is calculated and used as the feature-level distillation loss.

[0071] Feature-level distillation loss is added as a loss term in the quantization training process of the multi-task network and then used to train the multi-task network.

[0072] Furthermore, the expression for calculating the inverse energy score is as follows:

[0073]

[0074] in The inverse energy score represents the spatial location (i,j); The original image features are for spatial location (i,j); and These represent the mean and variance within the channel, respectively. This is the equilibrium constant.

[0075] Furthermore, the expressions for calculating probability distributions P and Q are as follows:

[0076]

[0077]

[0078] in This indicates Softmax normalization; e is the natural constant. Represents the enhanced image features A corresponding to the teacher network. f The i-th element; Represents the enhanced image features A corresponding to the teacher network. f The j-th element; The enhanced image feature A corresponding to the quantization model. q The i-th element; The enhanced image feature A corresponding to the quantization model. q The j-th element.

[0079] Furthermore, the difference between probability distributions P and Q can be calculated using JS divergence, KL divergence, and mean squared error; when the calculation method is JS divergence, the expression for the characteristic-level distillation loss is:

[0080]

[0081] in The characteristic-level distillation loss corresponding to the difference between probability distributions P and Q is calculated using JS divergence; Let (i,j) be the numerical value corresponding to the spatial location (i,j) in the probability distribution P; Let be the numerical value corresponding to the spatial location (i,j) in the probability distribution Q; It is half the sum of the corresponding values ​​of spatial location (i,j) in probability distributions P and Q, i.e. ; It represents the logarithm to the base 2.

[0082] The beneficial effects of this invention are as follows:

[0083] 1. This method optimizes the features that need to be fused and normalizes all channels of the fused features at each spatial location, thereby optimizing the balanced contribution of shallow and deep branches in the quantization training process of multi-task networks, reducing gradient bias in the quantization training process of multi-task networks, and improving the accuracy of multi-task networks after quantization training.

[0084] 2. This method is specifically designed for feature fusion structures in multi-task networks. It is applicable to any multi-task network that includes a feature fusion mechanism and can be easily integrated as a plug-in component. It is versatile and does not affect the original structure of the multi-task network.

[0085] 3. The attention distribution alignment loss strategy proposed in this method alleviates the feature gradient imbalance problem caused by the accumulation of quantization noise during backpropagation. The feature gradients are rescaled to a more uniform amplitude, thereby correcting the optimization bias and enabling more equitable treatment of features from different tasks during quantization.

[0086] 4. Through extensive experiments on various datasets and network architectures, this method has fully verified its effectiveness and robustness. It not only maintains stable performance improvement across different QAT algorithms but also demonstrates good generalization ability. Overall, its performance is significantly better than the current state-of-the-art quantization methods.

[0087] 5. This method has good mathematical interpretability. Attached Figure Description

[0088] Figure 1 This is a flowchart illustrating the method.

[0089] Figure 2 Diagram of the existing YOLOv5 network structure;

[0090] Figure 3 The average gradient magnitude measurement results for existing YOLOv5 feature fusion nodes;

[0091] Figure 4 A flowchart illustrating the workflow of the Attention Distribution Alignment Loss (ADA) strategy;

[0092] Figure 5 This represents the average gradient magnitude of the feature fusion node after using this method in Example 2. Detailed Implementation

[0093] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0094] Example 1:

[0095] like Figure 1 As shown, the method for reducing gradient bias during the quantization training process of multi-task networks includes the following steps:

[0096] S1. During the process of quantizing the multi-task network with the image as input, obtain the shallow and deep image features that need to be fused in the feature fusion layer of the multi-task network.

[0097] S2. Multiply the shallow features of the image with the first learnable weight and multiply the deep features of the image with the second learnable weight to optimize the balanced contribution of the shallow and deep branches in the multi-task network quantization training process.

[0098] S3. Use the results of the two multiplications as input to the corresponding feature fusion layer to obtain the fused image features;

[0099] S4. Normalize all channels of the fused image features at each spatial location to obtain normalized image features;

[0100] S5. The normalized image features are used as input to the subsequent structure of the feature fusion layer of the multi-task network to reduce gradient bias during the quantization training process of the multi-task network and improve the accuracy of the multi-task network after quantization training.

[0101] Steps S1 to S5 above can be summarized as a gradient-aware balanced feature fusion strategy.

[0102] In this embodiment, the expressions for multiplying the shallow image features by the first learnable weight and the deep image features by the second learnable weight in step S2 are as follows:

[0103]

[0104]

[0105] in shallow features of the image With the first learnable weight The result of multiplication; For deep image features With the second learnable weight The result of multiplication; and All The number between, and and The sum is 1. and The model can adaptively control and The fusion ratio. For each spatial location (i,j), the concatenated vector can be represented as... Its length is ,in and These represent the number of channels for shallow and deep image features, respectively.

[0106] In this embodiment, the specific method for normalizing all channels of the fused image features at each spatial location in step S4 to obtain the normalized image features includes:

[0107] Calculate the mean of the fused image features. and standard deviation Its expression is:

[0108]

[0109]

[0110] in and These represent the number of channels for shallow and deep image features, respectively. For channel indexing; For the fused image features in the first... Image features under a single channel;

[0111] The fused image features are normalized for each channel using the following expression:

[0112]

[0113] in For the normalized first Image features under a single channel;

[0114] By iterating through each channel, the normalized image features are obtained.

[0115] The process of normalizing all channels at each spatial location can be accomplished using the LayerNorm module, which is treated as a one-time plug-in component. This LayerNorm module normalizes all channels at each spatial location, effectively centering and scaling the gradient, similar to gradient whitening. It's important to note that these LayerNorm modules can be safely removed during the inference phase (after quantization training), as their primary function is to stabilize feature distribution and gradient flow during quantization training. Once the quantization parameters have been learned, they no longer contribute to the model's expressive power and performance during inference and can therefore be collapsed or omitted to reduce computational overhead.

[0116] Assuming the loss function of a multi-task network is L, according to the chain rule, the features before normalization... The gradient can be expressed as:

[0117] .

[0118] This indicates that the gradient of all channels is the same. Scale and subtract the same mean. The result is that gradients in shallow branches are relatively amplified, while gradients in deep branches are suppressed. Combined with learnable weights... By leveraging its advantages, this method enables the two branches to contribute equally during the optimization process, thereby promoting balanced representation learning.

[0119] In this embodiment, the multi-task network is a network that includes classification and regression tasks, used for object detection or image segmentation; wherein object detection includes image classification and localization.

[0120] Example 2:

[0121] From another perspective, the optimization bias observed in Example 1 stems from the incompatibility between traditional loss function design and quantized multi-task networks. Existing loss functions typically utilize perturbations caused by quantization parameters and their deviations from the true values ​​as supervision, aiming to reduce numerical discrepancies. However, due to the inherent nonlinearity and redundancy of neural networks, such low-level numerical discrepancies have limited impact on task performance because the model's predictions rely primarily on learned features rather than numerical bias.

[0122] Therefore, this embodiment is a further extension based on Embodiment 1. Specifically, this embodiment adds an attention distribution alignment loss strategy (ADA) to the gradient-aware balanced feature fusion strategy. The attention distribution alignment loss strategy includes the following operations:

[0123] A1. Use the pre-trained full-precision model as the teacher network and the multi-task network to be quantized as the quantization model.

[0124] A2. Using the same image as input to both the teacher network and the quantization model, the attention distribution is extracted from both the teacher network and the quantization model using the SimAM mechanism, and the mean and variance of each channel are calculated.

[0125] A3. The mean and variance obtained based on the SimAM (Simple, Parameter-Free Attention Module) mechanism define an inverse energy score for each spatial location of the input image;

[0126] A4. Map the inverse energy score through the sigmoid function to obtain the attention weights between 0 and 1 for each spatial location, and then obtain the attention distribution corresponding to the teacher network and the attention distribution corresponding to the quantization model.

[0127] A5. Weight the original image features of spatial location with the corresponding attention weights to obtain the enhanced image features, i.e., the enhanced image features A corresponding to the teacher network. f Enhanced image features A corresponding to the quantization model q ;

[0128] A6. Normalize the attention distribution corresponding to the teacher network and the attention distribution corresponding to the quantization model into probability distributions P and Q using Softmax, respectively. Calculate the difference between probability distributions P and Q, and use this difference as the feature-level distillation loss.

[0129] A7. Incorporate the feature-level distillation loss as a loss term in the multi-task network quantization training process into the total loss of the multi-task network, and train the multi-task network using the total loss of the multi-task network.

[0130] In this embodiment, the expression for calculating the inverse energy score is:

[0131]

[0132] in The inverse energy score represents the spatial location (i,j); The original image features are for spatial location (i,j); and These represent the mean and variance within the channel, respectively. This is the equilibrium constant.

[0133] The expressions for calculating probability distributions P and Q are:

[0134]

[0135]

[0136] in This indicates Softmax normalization; e is the natural constant. Represents the enhanced image features A corresponding to the teacher network. f The i-th element; Represents the enhanced image features A corresponding to the teacher network. f The j-th element; The enhanced image feature A corresponding to the quantization model. q The i-th element; The enhanced image feature A corresponding to the quantization model. q The j-th element.

[0137] The difference between probability distributions P and Q can be calculated using JS divergence, KL divergence, and mean square error. When the calculation method is JS divergence, the expression for the characteristic-level distillation loss is:

[0138]

[0139] in The characteristic-level distillation loss corresponding to the difference between probability distributions P and Q is calculated using JS divergence; Let (i,j) be the numerical value corresponding to the spatial location (i,j) in the probability distribution P; Let be the numerical value corresponding to the spatial location (i,j) in the probability distribution Q; It is half the sum of the corresponding values ​​of spatial location (i,j) in probability distributions P and Q, i.e. ; It represents the logarithm to the base 2.

[0140] like Figure 4 As shown ( Figure 4 In this model (where the DNN is the teacher network and the QNN is the quantization model), directly applying the original quantization method in the early stages of training will result in the loss of a significant amount of crucial attention information, especially in target region perception. As the distillation process progresses, the attention features of the quantization model are gradually recovered, thereby achieving more accurate localization and representation of the target region.

[0141] like Figure 5 As shown, with Figure 3 In contrast, applying this method alleviates the imbalance of feature gradients caused by the accumulation of quantization noise during backpropagation. The feature gradients are rescaled to a more uniform amplitude, thereby correcting the imbalance. Figure 3 The optimization bias observed in the data allows for a fairer treatment of features from different tasks during the quantization process. This result can also be explained using the formula for the loss function L mentioned above.

[0142] This method not only maintains stable performance improvements across different QAT algorithms but also demonstrates excellent generalization ability, with overall results significantly outperforming state-of-the-art quantization methods. Experiments on the two major datasets, PASCAL VOC and MS COCO, further confirm this: under different network structures and bit widths, our method (Ours) achieves an average mAP improvement of approximately 3.3% and 1.6% respectively compared to various state-of-the-art quantization strategies (as shown in Tables 2 and 3). Particularly noteworthy is that in the 4-bit quantization scenario of YOLOv5, our method reduces the accuracy gap between the quantized model and the full-precision model to only 1.7%. This result fully demonstrates that our method can maintain near-full-precision performance even under low-bit constraints, showcasing its potential and advantages in practical deployment scenarios.

[0143] Table 2: Results of this method on the VOC dataset

[0144]

[0145] Table 3: Results of this method on the COCO dataset

[0146]

[0147] It should be noted that this gradient-aware balanced feature fusion strategy can achieve a stable improvement in mAP (i.e., improve the accuracy of the multi-task network after quantization training in object detection or image segmentation). The main advantage of the attention distribution alignment loss strategy is that it can stabilize training and help the quantized model converge quickly, which is equivalent to reducing the computational cost. It should also be noted that, as shown in Tables 2 and 3, although this method differs from the traditional training method for multi-task networks, it does not affect other aspects of the traditional training method, such as the use of other loss functions and network parameter optimization methods.

[0148] In this embodiment, to analyze the gradient scaling effect of Layer Normalization (LN) on the input features, we first define the forward computation of a single sample. Let the concatenated feature vector be h = , where D = This indicates the total number of channels after splicing. , These are the number of channels from the two input branches, respectively. The forward propagation of LN is defined as:

[0149]

[0150]

[0151] in Represents the j-th input feature. This represents the normalized output. The mean, The standard deviation is given. The stability term was omitted in the gradient derivation. (A very small value to avoid division by zero).

[0152] The partial derivatives of the mean and standard deviation with respect to the input are:

[0153]

[0154] Next, calculate about The derivative:

[0155]

[0156] in It is Kronecker delta, and .

[0157] set up This represents the gradient from the loss function. Definition:

[0158]

[0159] Applying the chain rule, we can obtain the complete gradient of LN with respect to the input:

[0160]

[0161] General assumptions and If they are approximately uncorrelated in terms of expectation, then The above formula can be simplified to:

[0162]

[0163] This corresponds to the commonly used "centering + uniform scaling" form of gradient representation (equivalent to gradient whitening, and uniform centering and scaling will relatively reduce gradient bias). Essentially, it is what was mentioned earlier. .

[0164] In summary, this invention provides the first in-depth analysis of the fundamental reasons for the performance degradation of multi-task networks under low-bit quantization, pointing out that feature fusion mismatch and gradient optimization bias are key bottlenecks in the quantization-aware training process. Therefore, the gradient-aware balancing strategy and attention distribution alignment strategy proposed in this method are the first to address this problem. Their core is a concise and efficient solution to the identified gradient bias problem. Their advantage lies not in architectural complexity, but in the precise matching between the problem, mechanism, and solution. Furthermore, this method possesses good mathematical interpretability.

Claims

1. A method for reducing gradient bias in a multi-task network quantization training process, the method comprising: Includes the following steps: In the process of quantizing and training a multi-task network with images as input, the shallow and deep features of the image that need to be fused in the feature fusion layer of the multi-task network are obtained. The shallow image features are multiplied by the first learnable weight, and the deep image features are multiplied by the second learnable weight to optimize the balanced contribution of the shallow and deep branches in the multi-task network quantization training process. The results of the two multiplications are used as input to the corresponding feature fusion layer to obtain the fused image features; The fused image features are normalized for all channels at each spatial location to obtain normalized image features; Normalized image features are used as input to the subsequent structure of the feature fusion layer of the multi-task network to reduce gradient bias during the quantization training process and improve the accuracy of the multi-task network after quantization training.

2. The method of claim 1, wherein, The expression for multiplying shallow image features by the first learnable weight and deep image features by the second learnable weight is as follows: in shallow features of the image With the first learnable weight The result of multiplication; For deep image features With the second learnable weight The result of multiplication.

3. The method of claim 2, wherein, and The sum is 1.

4. The method for reducing gradient bias in multi-task network quantization training process according to claim 1, wherein, The specific methods for normalizing all channels of the fused image features at each spatial location to obtain normalized image features include: calculating a mean value of the fused image features and a standard deviation whose expression is: in and These represent the number of channels for shallow and deep image features, respectively. For channel indexing; For the fused image features in the first... Image features under a single channel; The fused image features are normalized for each channel using the following expression: wherein is the normalized first image feature under the channel; By iterating through each channel, the normalized image features are obtained.

5. The method for reducing gradient bias in multi-task network quantization training process according to claim 1, wherein, Multi-task networks are networks that include classification and regression tasks, used for object detection or image segmentation; object detection includes image classification and localization.

6. The method for reducing gradient bias in multi-task network quantization training process according to claim 1, wherein, The following operations are also included in the quantization training process of multi-task networks: The pre-trained full-precision model is used as the teacher network, and the multi-task network to be quantized and trained is used as the quantization model. The same image is used as input to both the teacher network and the quantization model. The attention distribution is extracted from the teacher network and the quantization model respectively through the SimAM mechanism, and the mean and variance of each channel are calculated. The mean and variance obtained based on the SimAM mechanism define an inverse energy score for each spatial location of the input image; The inverse energy score is mapped through the sigmoid function to obtain the attention weights between 0 and 1 at each spatial location, thereby obtaining the attention distribution corresponding to the teacher network and the attention distribution corresponding to the quantization model. The original image features of spatial location are weighted with their corresponding attention weights to obtain the enhanced image features, i.e., the enhanced image features A corresponding to the teacher network. f Enhanced image features A corresponding to the quantization model q ; The attention distribution corresponding to the teacher network and the attention distribution corresponding to the quantization model are normalized to probability distributions P and Q using Softmax, respectively. The difference between probability distributions P and Q is calculated and used as the feature-level distillation loss. Feature-level distillation loss is added as a loss term in the quantization training process of the multi-task network and then used to train the multi-task network.

7. The method of claim 6, wherein, The expression for calculating the inverse energy score is: in The inverse energy score represents the spatial location (i,j); The original image features are for spatial location (i,j); and These represent the mean and variance within the channel, respectively. This is the equilibrium constant.

8. The method for reducing gradient bias during quantization training of multi-task networks according to claim 6, characterized in that, The expressions for calculating probability distributions P and Q are: in This indicates Softmax normalization; e is the natural constant. Represents the enhanced image features A corresponding to the teacher network. f The i-th element; Represents the enhanced image features A corresponding to the teacher network. f The j-th element; The enhanced image feature A corresponding to the quantization model. q The i-th element; The enhanced image feature A corresponding to the quantization model. q The j-th element.

9. The method for reducing gradient bias in a multi-task network quantization training process according to claim 6, wherein, The difference between probability distributions P and Q can be calculated using JS divergence, KL divergence, and mean square error. When the calculation method is JS divergence, the expression for the characteristic-level distillation loss is: in The characteristic-level distillation loss corresponding to the difference between probability distributions P and Q is calculated using JS divergence; Let (i,j) be the numerical value corresponding to the spatial location (i,j) in the probability distribution P; Let be the numerical value corresponding to the spatial location (i,j) in the probability distribution Q; It is half the sum of the corresponding values ​​of spatial location (i,j) in probability distributions P and Q, i.e. ; It represents the logarithm to the base 2.

Images